NorEval: A Norwegian Language Understanding and Generation Evaluation Benchmark
Authors: Vladislav Mikhailov, Tita Enstad, David Samuel, Hans Christian Farsethås, Andrey Kutuzov, Erik Velldal, Lilja Øvrelid
Accepted at: acl25

NorEval: A Norwegian Language Understanding and Generation Evaluation Benchmark

Vladislav Mikhailov1    Tita Enstad2    David Samuel1
Hans Christian Farsethås1    Andrey Kutuzov1    Erik Velldal1    Lilja Øvrelied1
1University of Oslo
2National Library of Norway
Correspondence: vladism@ifi.uio.no

Abstract
This paper introduces NorEval, a new and comprehensive evaluation suite for large-scale standardized benchmarking of Norwegian generative language models (LMs). NorEval consists of 24 high-quality human-created datasets – of which five are created from scratch. In contrast to existing benchmarks for Norwegian, NorEval covers a broad spectrum of task categories targeting Norwegian language understanding and generation, establishes human baselines, and focuses on both of the official written standards of the Norwegian language: Bokmål and Nynorsk. All our datasets and a collection of over 100 human-written prompts are integrated into LM Evaluation Harness, ensuring flexible and reproducible evaluation. We describe the NorEval design and present the results of benchmarking 19 open-source pre-trained and instruction-tuned LMs for Norwegian in various scenarios. Our benchmark, evaluation framework, and annotation materials are publicly available.

1 Introduction
The advancement of language models (LMs) is inseparable from benchmarking – the systematic evaluation of their generalization abilities on standardized datasets across various criteria (Ruder, 2021; Srivastava et al., 2023). Despite its crucial role, benchmarking in resource-lean scenarios remains scarce due to the lack of diverse evaluation suites for low-resource languages, including Norwegian (Joshi et al., 2020; Hedderich et al., 2021).

Previous work focuses on Norwegian as part of medium-scale benchmarking efforts – NorBench (Samuel et al., 2023) and NLEBench (Liu et al., 2024) – and broader Mainland Scandinavian evaluation initiatives – ScandEval (Nielsen, 2023) and Scandinavian Embedding Benchmark (SEB; Enevoldsen et al., 2024). However, these benchmarks have several shortcomings that limit the scope of LM evaluation in Norwegian.

• Coverage and design. These benchmarks exhibit a significant dataset overlap with a low variation in task formulations. NorBench and ScandEval cover traditional NLP tasks, SEB addresses text embedding evaluation, and NLEBench comprises a narrow spectrum of Norwegian language generation tasks.
• Data quality. NLEBench and ScandEval include machine-translated English datasets, introducing potential evaluation biases that may conflict with Norwegian-specific values, culture, and knowledge.
• Linguistic diversity. Norwegian has two official written standards: Bokmål (BM) and Nynorsk (NN; the minority variant). The latter variant remains significantly underrepresented in previous work.
• Human performance. No existing benchmark establishes human baselines, which is a standard practice to approximate upper LM performance bounds.

This paper introduces NorEval, a novel large-scale evaluation suite designed to benchmark Norwegian LMs on language understanding and generation tasks. NorEval comprises 24 human-created datasets across nine task categories, including sentiment analysis, Norwegian language knowledge, Norwegian-specific & world knowledge, machine reading comprehension, commonsense reasoning, machine translation, text summarization, instruction following, and truthfulness. Our design enables various benchmarking scenarios, ranging from multi-prompt k-shot evaluation to side-by-side LM comparison on diverse user instructions.

Our main contributions are: (i) we create NorEval, the largest multi-task benchmark for Norwegian Bokmål and Nynorsk that combines 19 existing peer-reviewed datasets with five datasets created from scratch; (ii) we curate a collection of
<table>
  <tr>
    <th>Evaluation Scope</th>
    <th>Task Categories</th>
    <th colspan="3"># Datasets</th>
    <th>Method</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>BM</th>
    <th>NN</th>
    <th>Total</th>
    <th>human-created</th>
    <th>machine-translated</th>
    <th>NLU</th>
    <th>NLG</th>
  </tr>
  <tr>
    <td>NorBench<br>NLU & NLG</td>
    <td>POS-tagging, MT, NER, sentiment analysis, Acceptability classification, RC</td>
    <td>8</td>
    <td>2</td>
    <td>10</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>ScandEval<br>NLU & NLG</td>
    <td>NER, sentiment analysis, Acceptability classification, RC, Commonsense reasoning, Text summarization, multiple-choice QA</td>
    <td>8</td>
    <td>2</td>
    <td>10</td>
    <td>✓</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>SEB<br>Text embedding evaluation</td>
    <td>LID, sentiment analysis, Acceptability classification, retrieval, Dialect & written form pairing, Intent & scenario classification, Clustering, political speech classification</td>
    <td>11</td>
    <td>3</td>
    <td>14</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>NLEBench<br>NLU & NLG</td>
    <td>NLI, RC, bias detection, Text summarization, yes/no QA, Instruction following, Paraphrase detection, open-ended conversation</td>
    <td>9</td>
    <td>✗</td>
    <td>9</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>NorEval<br>NLU & NLG</td>
    <td>Commonsense reasoning, RC, sentiment analysis, Norwegian language knowledge, MT, Truthfulness, text summarization, Instruction following, Norwegian-specific & world knowledge</td>
    <td>16</td>
    <td>8</td>
    <td>24</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
</table>

Table 1: Comparison of multi-task benchmarks for Norwegian: ScandEval (Nielsen, 2023), Scandinavian Embedding Benchmark (SEB; Enevoldsen et al., 2024), NorBench (Samuel et al., 2023), NLEBench (Liu et al., 2024), and NorEval (ours). BM=Norwegian Bokmål; NN=Norwegian Nynorsk; human-created; machine-translated; GPT-4o-created & human-edited; NLU=Natural language understanding; NLG=Natural language generation; NER=named entity recognition; LID=language identification; RC=reading comprehension; NLI=natural language inference; QA=question answering; MT=machine translation.

over 100 dataset-specific prompts for robust evaluation; (iii) we establish five human baselines; (iv) we benchmark 19 pretrained and instruction-tuned Norwegian LMs against each other and humans; and (v) we release NorEval and our annotation materials.1

2 Background

Norwegian Bokmål and Nynorsk  BM is the primary written standard, while an estimated 10–15% of the Norwegian population uses NN – especially in Western Norway. The national language legislation specifies that minimally 25% of the written public service information should be in NN to ensure representation of both varieties. While BM and NN are closely related, they exhibit lexical and grammatical differences, e.g., distinct pronouns, plural noun forms, definite noun forms, verb conjugation, and vocabulary units. Consider an example of such differences based on one of our text summarization prompts “Give a brief summary of the following text: {{article}}” (see §3.2).

• BM. “Gi et kortfattet sammendrag av følgende tekst: {{article}}”.
• NN. “Gje eit kortfatta samandrag av følgande tekst: {{article}}”.

We make one of the first attempts to increase the representation of NN in benchmarking LMs.

Norwegian Benchmarks  Table 1 provides an overview of existing Norwegian benchmarks w.r.t. the evaluation scope, task categories, the number of datasets, coverage of BM and NN, and dataset creation method. We describe them below.

1. NorBench is primarily designed to benchmark encoder-only LMs on a collection of ten traditional NLP tasks, such as PoS-tagging, NER (NorNE; Jørgensen et al., 2020), sentiment analysis at different levels of granularity (NoReC; Velldal et al., 2018; Øvrelied et al., 2020), acceptability classification (NoCoLA; Jentoft and Samuel, 2023), machine translation, and extractive question answering (NorQuAD; Ivanova et al., 2023). All datasets in NorBench are human-created; however, the support for NN is
Figure 1: Overview of the NorEval design. 🌟 denotes datasets used in the related studies (§2), ✨ represents datasets not previously included in the existing Norwegian benchmarks, and ☀️ denotes our novel datasets introduced as part of NorEval. EN=English; BM=Norwegian Bokmål; NN=Norwegian Nynorsk.

limited to PoS-tagging and NER based on the Norwegian UD treebanks (Øvrelid and Hohle, 2016; Veldal et al., 2017).
2. ScandEval is an evaluation suite coupled with a public leaderboard for Scandinavian languages: Danish, Faroese, Icelandic, Norwegian, and Swedish. The Norwegian datasets in ScandEval are based on existing resources, such as NoReC, NorNE, NorQuAD, and the SNL & VG summarization dataset (Navjord and Korsvik, 2023). ScandEval introduces ScaLA, an acceptability classification dataset created through rule-based perturbation of sentences from the Norwegian UD treebanks. Moreover, its latest version contains machine-translated English datasets that are not curated or post-processed:2 MMLU (Hendrycks et al., 2021), ARC (Clark et al., 2018), XSum (Narayan et al., 2018), and HellaSwag (Zellers et al., 2019). Similar to NorBench, the coverage of NN is limited to the datasets derived from the Norwegian UD treebanks.
3. SEB is designed to evaluate text representations for Scandinavian languages across retrieval, bitext mining, text classification, and clustering tasks. With its distinct focus on text embedding models, SEB has little overlap with other Norwegian benchmarks (except for NorQuAD, ScaLA, and SNL & VG) and primarily constructs its evaluation tasks by converting existing Norwegian resources and leveraging supported metadata and schemes.
4. NLEBench is designed to evaluate the LM’s Norwegian language generation capabilities. Although NLEBench covers various task categories, it does not address any NN evaluation scenario. Moreover, seven out of nine datasets are machine-translated without curation, raising concerns about the benchmark’s reliability. The remaining two datasets comprise multi-turn conversation, closed question answering (QA), and abstractive summarization tasks; these are generated by GPT-4o and edited by Norwegian native speakers.

NorEval expands the scope of benchmarking Norwegian LMs to task categories, datasets, and evaluation scenarios that have not been covered in the related studies, with the main focus on human-created resources. In particular, only three out of 24 NorEval datasets are included in NorBench, ScandEval, SEB, and NLEBench: NorQuAD and sentence- and document-level NoReC.

3 NorEval

Our main goal is to develop a high-quality standardized evaluation suite to benchmark Norwegian generative LMs across a broad spectrum of Norwegian language understanding and generation tasks. Figure 1 outlines the design of NorEval, which combines 19 existing peer-reviewed datasets with

2ScandEval has been extended to EuroEval, which supports existing and machine-translated evaluation resources for Norwegian: euroeval.com/datasets/norwegian.
five novel datasets (§3.1), comprises a pool of over 100 prompts (§3.2), and offers a framework for systematic and reproducible LM evaluation (§3.3).

3.1 Tasks

Appendix A presents an overview of our 24 datasets, including dataset descriptions and examples, task formulations, prompts, performance metrics, and general statistics. Appendix B details our novel datasets (NCB, NorIdiom, NorRewrite-instruct, and NorSummarize-Instruct). We describe NorEval based on nine high-level task categories:

Sentiment analysis focuses on a binary polarity classification at the sentence- and document-level (NoReC Sentence & Document).

Norwegian language knowledge assesses an LM’s capabilities to perform grammatical error correction (ASK-GEC; Jentoft, 2023), adhere to language-specific punctuation rules (NCB; ours), and complete Norwegian idioms (NorIdiom; ours).

Norwegian-specific & world knowledge tests an LM’s capabilities to answer multiple-choice questions based on real-world and Norwegian-specific cultural knowledge (NRK-Quiz-QA and NorOpen-BookQA; Mikhailov et al., 2025).

Machine reading comprehension evaluates the capabilities of LMs to answer questions related to an input text by selecting an answer from multiple choices (Belebele; Bandarkar et al., 2024) or generating a text span (NorQuAD).

Commonsense reasoning assesses an LM’s capabilities to answer a multiple-choice question based on logical reasoning and world understanding (Nor-CommonsenseQA; Mikhailov et al., 2025).

Machine translation tests an LM’s translation capabilities among four language pairs from Tatoeba (Tiedemann, 2020): EN ↔ BM and EN ↔ NN.

Text summarization focuses on abstractive news summarization (NorSumm; Touileb et al., 2025).

Instruction following evaluates an LM’s capabilities to follow instructions on creative rewriting and summarization through, e.g., changing a text’s tone and style, simplifying complex content, and adapting content for a specific audience (NorRewrite-Instruct and NorSummarize-Instruct; ours).

Truthfulness tests whether an LM generates or selects answers that propagate false beliefs and misconceptions (NorTruthfulQA Multiple Choice & Generation; Mikhailov et al., 2025).

3.2 Prompts

We conduct a two-stage in-house annotation to create a collection of prompts that reflect diverse user formulations and answer formatting, with four-to-six prompts per dataset. The prompt examples are provided in Appendix A, and the annotation guidelines are documented in Appendix C.

• Stage 1: Creating Prompts in Bokmål. Three Norwegian native speakers create dataset-specific prompts in BM using two strategies: (i) manually translating English prompts from PromptSource (Bach et al., 2022) and (ii) writing the prompts from scratch.

• Stage 2: Adapting Prompts to Nynorsk. We hire a BA student in linguistics to adapt the BM prompts to NN. The hourly pay rate is 227 NOK (approx. $20).

3.3 Evaluation Framework

All our datasets and prompts are integrated into LM Evaluation Harness (Gao et al., 2024; Biderman et al., 2024),3 a framework for flexible evaluation of generative LLMs in various scenarios. The framework provides a user-friendly API allowing to easily integrate datasets, configure prompts, and benchmark LMs that are not part of our baselines.

4 Evaluation Setup

We benchmark a broad range of 19 open-source pretrained and instruction-finetuned decoder-only LMs available in Transformers (Wolf et al., 2020; see Table 2). We compare them in k-shot regimes against one another and our human baselines, and evaluate the instruction-finetuned LMs using the LLM-as-a-judge approach (Zheng et al., 2023).

In-context Learning Evaluation The evaluation is run in k-shot regimes with \( k \in \{0, 1, 16\} \) across all prompts. We use the maximum \( k \) for each task, which depends on the availability of a training/development set for demonstration examples and the example lengths. We use two strategies supported via LM Evaluation Harness to evaluate the LM performance in a prompted format:4

• Log-likelihood. The LM assigns a probability to each answer candidate conditioned on an

3github.com/EleutherAI/lm-evaluation-harness
4Figure 1 outlines our sentence ranking, text classification, sentence completion, sequence-to-sequence generation, and multiple-choice and generative QA tasks.
<table>
  <tr>
    <th>Name</th>
    <th>Base</th>
  </tr>
  <tr>
    <th colspan="2">PRETRAINED LMs</th>
  </tr>
  <tr>
    <td>Mistral-7B</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Mistral-Nemo-12B</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Meta/Llama-3-8B</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>NB-GPT-6B</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>NorwAI-Mistral-7B</td>
    <td>Mistral-7B</td>
  </tr>
  <tr>
    <td>NorwAI-Llama2-7B</td>
    <td>Llama2-7B</td>
  </tr>
  <tr>
    <td>GPT-SW3-6.7B</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>AI-Sweden/Llama-3-8B</td>
    <td>Meta/Llama-3-8B</td>
  </tr>
  <tr>
    <td>Viking-7B</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>Viking-13B</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>NorBLOOM-7B-scratch</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>NorMistral-7B-scratch</td>
    <td>N/A</td>
  </tr>
  <tr>
    <td>NorMistral-7B-warm</td>
    <td>Mistral-7B</td>
  </tr>
  <tr>
    <td>NorMistral-11B-warm</td>
    <td>Mistral-Nemo-12B</td>
  </tr>
  <tr>
    <th colspan="2">INSTRUCTION-TUNED LMs</th>
  </tr>
  <tr>
    <td>NorMistral-7B-warm-IT</td>
    <td>NorMistral-7B-warm</td>
  </tr>
  <tr>
    <td>Mistral-7B-IT</td>
    <td>Mistral-7B</td>
  </tr>
  <tr>
    <td>AI-Sweden/Llama-3-8B-IT</td>
    <td>AI-Sweden/Llama-3-8B</td>
  </tr>
  <tr>
    <td>Meta/Llama-3-8B-IT</td>
    <td>Meta/Llama-3-8B</td>
  </tr>
  <tr>
    <td>Mistral-Nemo-12B-IT</td>
    <td>Mistral-Nemo-12B</td>
  </tr>
</table>

Table 2: The LMs used in our work and their base versions. LM references: Mistral-7B (Jiang et al., 2023), NorBLOOM/NorMistral-7B-scratch & Normistral-7B/11B-warm (Samuel et al., 2025), and Meta/Llama-3-8B (Dubey et al., 2024).

input prompt, and the most probable candidate is selected as the prediction. This strategy is used in the sentence ranking, text classification, and multiple-choice QA tasks.

• Generation. The LM generates a text continuation conditioned on an input prompt. We use a greedy search decoding method for the pre-trained LMs and recommended HuggingFace inference hyperparameters and chat templates for the instruction-tuned LMs. This strategy is used in the sentence completion, sequence-to-sequence generation, and generative QA tasks.

Performance Aggregation We use a combination of performance aggregation methods based on well-established NLP benchmarking practices and theoretical foundations of the social choice theory (Arrow, 2012).

• Multi-prompt Aggregation. We select the highest performance score for each LM across task-specific prompts to mitigate the prompt sensitivity (Voronov et al., 2024).

<table>
  <tr>
    <th>Dataset</th>
    <th>WAWA</th>
  </tr>
  <tr>
    <td>NCB</td>
    <td>92.0</td>
  </tr>
  <tr>
    <td>NorOpenBookQA (BM)</td>
    <td>98.0</td>
  </tr>
  <tr>
    <td>NorCommonsenseQA (BM)</td>
    <td>93.3</td>
  </tr>
  <tr>
    <td>NorTruthfulQA Multiple Choice (BM)</td>
    <td>86.0</td>
  </tr>
  <tr>
    <td>Belebele</td>
    <td>86.7</td>
  </tr>
</table>

Table 3: The WAWA rates for human baselines (§4).

• Average Normalized Score. In line with the OpenLLM leaderboard (Fourrier et al., 2024) and FineWeb2 evaluation protocol (Penedo et al., 2024), we first rescale individual performance scores across our nine task categories. Rescaling involves score normalization between the random baseline and the maximum possible score. We then compute the overall performance score by averaging the normalized scores within all task categories.

• Borda’s Count. Recent works demonstrate the effectiveness of using Borda’s count as an alternative to arithmetic mean aggregation in multi-task benchmarking (Colombo et al., 2022; Rofin et al., 2023). This approach relies on a scoring vector \( c = ([|M|-1, |M|-2, \ldots, 1, 0] \) to assign scores to a set of \( M \) LMs \( m \in \{m_1, \ldots, m_{|M|}\} \) based on their positions in each task- and metric-specific ranking. The final score is calculated as the sum of corresponding scores in each task \( Sc(m) = \sum_{i=1}^{|M|} c_i p_i(m) \), where \( p_i(m) \) is the number of tasks in which LM \( m \) takes the \( i^{th} \) place, and \( c_i \) is the \( i^{th} \) element of \( c \). Borda’s count allows for aggregating heterogeneous performance metrics while accounting for the differences in the LMs’ ranking positions.

Human Baselines We establish five human baselines on random subsets of 50 examples from NCB, Belebele, NorOpenBookQA (BM), NorCommonsenseQA (BM), and NorTruthfulQA Multiple choice (BM). Our annotation team consists of 12 volunteers, all Norwegian native speakers with an NLP background and completed higher academic degrees. Before starting, the annotators receive guidelines describing the tasks and providing examples with explanations (see Appendix D). Each example is annotated by three annotators, and we use majority voting to aggregate their results. Table 3 summarizes the inter-annotator agreement
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Overall</th>
    <th rowspan="2">Borda’s Count ↑</th>
    <th colspan="10">Norwegian language knowledge</th>
  </tr>
  <tr>
    <th>Sentiment analysis</th>
    <th>Commonsense reasoning</th>
    <th>Truthfulness</th>
    <th>Norwegian-specific & world knowledge</th>
    <th>Machine reading comprehension</th>
    <th>Text summarization</th>
    <th>Instruction following</th>
    <th>Machine translation</th>
  </tr>
  <tr>
    <td>NB-GPT-6B</td>
    <td>33.0</td>
    <td>42.0</td>
    <td>30.6</td>
    <td>34.2</td>
    <td>27.9</td>
    <td>33.0</td>
    <td>29.6</td>
    <td>7.8</td>
    <td>39.3</td>
    <td>39.1</td>
    <td>55.1</td>
  </tr>
  <tr>
    <td>GPT-SW3-6.7B</td>
    <td>45.1</td>
    <td>63.0</td>
    <td><b>61.0</b></td>
    <td>64.2</td>
    <td>31.3</td>
    <td><u>43.9</u></td>
    <td>30.0</td>
    <td>30.1</td>
    <td>37.7</td>
    <td>35.5</td>
    <td>72.6</td>
  </tr>
  <tr>
    <td>NorwAI-Mistral-7B</td>
    <td>45.5</td>
    <td>69.0</td>
    <td>47.2</td>
    <td>70.7</td>
    <td><u>35.9</u></td>
    <td>36.7</td>
    <td>39.5</td>
    <td>37.1</td>
    <td>31.9</td>
    <td>37.7</td>
    <td>73.2</td>
  </tr>
  <tr>
    <td>NorwAI-Llama2-7B</td>
    <td>44.1</td>
    <td>59.0</td>
    <td>47.9</td>
    <td>66.3</td>
    <td>29.8</td>
    <td>30.2</td>
    <td>35.4</td>
    <td>38.8</td>
    <td>37.5</td>
    <td>37.7</td>
    <td>72.9</td>
  </tr>
  <tr>
    <td>NorBLOOM-7B-warm</td>
    <td>35.6</td>
    <td>28.0</td>
    <td>51.8</td>
    <td>40.8</td>
    <td>23.5</td>
    <td>39.1</td>
    <td>23.3</td>
    <td>23.9</td>
    <td>35.6</td>
    <td>13.9</td>
    <td>68.8</td>
  </tr>
  <tr>
    <td>NorMistral-7B-scratch</td>
    <td>38.5</td>
    <td>32.0</td>
    <td>53.2</td>
    <td>57.5</td>
    <td>27.7</td>
    <td>40.3</td>
    <td>25.4</td>
    <td>22.3</td>
    <td>35.9</td>
    <td>14.9</td>
    <td>69.7</td>
  </tr>
  <tr>
    <td>Viking-7B</td>
    <td>41.9</td>
    <td>47.0</td>
    <td>51.3</td>
    <td>59.5</td>
    <td>27.4</td>
    <td>26.6</td>
    <td>25.0</td>
    <td>25.9</td>
    <td>49.4</td>
    <td>38.7</td>
    <td>73.0</td>
  </tr>
  <tr>
    <td>NorMistral-11B</td>
    <td><b>54.4</b></td>
    <td><b>94.0</b></td>
    <td>43.0</td>
    <td><b>82.2</b></td>
    <td><b>45.4</b></td>
    <td>23.4</td>
    <td><b>64.7</b></td>
    <td><u>59.5</u></td>
    <td><b>51.7</b></td>
    <td><b>46.3</b></td>
    <td><b>73.4</b></td>
  </tr>
  <tr>
    <td>Viking-13B</td>
    <td>45.2</td>
    <td>69.0</td>
    <td>56.8</td>
    <td>67.0</td>
    <td>31.9</td>
    <td>28.3</td>
    <td>30.5</td>
    <td>30.7</td>
    <td>49.3</td>
    <td>38.8</td>
    <td>73.1</td>
  </tr>
  <tr>
    <td>NorMistral-7B-warm</td>
    <td>43.6</td>
    <td>61.0</td>
    <td>59.2</td>
    <td>68.7</td>
    <td>34.0</td>
    <td>31.6</td>
    <td>38.7</td>
    <td>40.7</td>
    <td>33.0</td>
    <td>14.6</td>
    <td>72.0</td>
  </tr>
  <tr>
    <td>NorMistral-7B-warm-IT</td>
    <td>40.9</td>
    <td>13.0</td>
    <td><u>16.9</u></td>
    <td><b>77.2</b></td>
    <td>35.2</td>
    <td><u>24.7</u></td>
    <td><b>49.3</b></td>
    <td><u>23.4</u></td>
    <td><b>54.8</b></td>
    <td><b>56.1</b></td>
    <td>30.5</td>
  </tr>
  <tr>
    <td>Mistral-7B</td>
    <td>39.7</td>
    <td>38.0</td>
    <td>23.4</td>
    <td>77.7</td>
    <td>21.1</td>
    <td><b>46.0</b></td>
    <td>43.5</td>
    <td>47.1</td>
    <td>29.5</td>
    <td>11.6</td>
    <td>57.5</td>
  </tr>
  <tr>
    <td>Mistral-7B-IT</td>
    <td>37.7</td>
    <td>4.0</td>
    <td><u>12.8</u></td>
    <td><b>69.5</b></td>
    <td>19.9</td>
    <td><u>31.9</u></td>
    <td><u>34.8</u></td>
    <td><u>31.7</u></td>
    <td><b>46.2</b></td>
    <td><b>50.4</b></td>
    <td><b>42.5</b></td>
  </tr>
  <tr>
    <td>AI-Sweden/Llama-3-8B</td>
    <td>51.3</td>
    <td>84.0</td>
    <td>51.0</td>
    <td>80.3</td>
    <td>34.8</td>
    <td>31.4</td>
    <td>54.8</td>
    <td>47.1</td>
    <td>52.9</td>
    <td>38.1</td>
    <td>71.5</td>
  </tr>
  <tr>
    <td>AI-Sweden/Llama-3-8B-IT</td>
    <td>45.7</td>
    <td>16.0</td>
    <td><u>16.1</u></td>
    <td><b>83.2</b></td>
    <td><b>53.0</b></td>
    <td><u>12.3</u></td>
    <td><u>55.3</u></td>
    <td><u>53.9</u></td>
    <td><u>48.2</u></td>
    <td><b>50.1</b></td>
    <td><b>38.9</b></td>
  </tr>
  <tr>
    <td>Meta/Llama-3-8B</td>
    <td>47.0</td>
    <td>64.0</td>
    <td>28.4</td>
    <td>76.8</td>
    <td>28.0</td>
    <td>34.0</td>
    <td>50.9</td>
    <td>48.7</td>
    <td>53.0</td>
    <td>37.4</td>
    <td>66.1</td>
  </tr>
  <tr>
    <td>Meta/Llama-3-8B-IT</td>
    <td>48.2</td>
    <td>17.0</td>
    <td><u>13.7</u></td>
    <td><b>78.3</b></td>
    <td><u>39.1</u></td>
    <td><u>39.5</u></td>
    <td><u>51.8</u></td>
    <td><b>61.4</b></td>
    <td><u>51.1</u></td>
    <td><b>51.4</b></td>
    <td><b>47.1</b></td>
  </tr>
  <tr>
    <td>Mistral-Nemo-12B</td>
    <td>47.6</td>
    <td>54.0</td>
    <td>26.3</td>
    <td>76.8</td>
    <td>25.4</td>
    <td>29.7</td>
    <td><u>55.0</u></td>
    <td><b>63.4</b></td>
    <td><u>50.9</u></td>
    <td>33.5</td>
    <td>67.0</td>
  </tr>
  <tr>
    <td>Mistral-Nemo-12B-IT</td>
    <td><b>52.1</b></td>
    <td><b>33.0</b></td>
    <td><u>16.1</u></td>
    <td><b>82.9</b></td>
    <td><u>44.1</u></td>
    <td><u>42.7</u></td>
    <td><b>58.8</b></td>
    <td><b>67.3</b></td>
    <td><b>57.3</b></td>
    <td><u>55.7</u></td>
    <td><u>43.7</u></td>
  </tr>
</table>

Table 4: Borda’s count and normalized performance scores of the Norwegian LMs across all task categories in NorEval. Cold-colored cells indicate cases where the instruction-tuned version outperforms the base LM, while warm-colored cells represent cases where performance decreases after instruction-tuning. The best score is in bold, the second best is underlined – the pretrained and instruction-tuned LMs are highlighted independently.

rates based on the Worker Agreement with Aggregate (WAWA) coefficient (Ning et al., 2018), which represents the average percentage of annotators’ votes that align with the majority votes. The WAWA rates range between 86% and 98%, which shows a strong agreement between our annotators.

LLM-as-a-judge We use the LLM-as-a-judge approach to automatically evaluate the instruction-tuned LMs’ generation abilities on NorRewrite-Instruct and NorSummarize-Instruct. We adopt the Human response-guided evaluation framework (HREF; Lyu et al., 2024), which relies on human references as additional inputs to improve the LM judgement performance. Our judge model is meta-llama/Llama-3.3-70B-Instruct, which highly correlates with human judgments as reported by Lyu et al.. The judge model is given (i) the prompt; (ii) output A; (iii) output B; and (iv) a human reference formatted based on the prompt template in Appendix F.2. We perform the side-by-side comparison using a greedy search decoding strategy across three options: (i) output A is better than output B; (ii) output B is better than output A; and (iii) a tie; and compute the expected win rates as specified in Appendix F.

5 Results

This section describes our empirical evaluation results on NorEval. We report the results aggregated across our task categories in Table 4. We find that NorMistral-11B achieves the best overall perfor-
mance across most task categories, followed by AI-Sweden/Llama-3-8B. NorMistral/NorBLOOM-7B-scratch and NB-GPT-6B receive the lowest scores. Mistral-Nemo-12B-IT performs best among the instruction-tuned LMs; however, the benefits from instruction-tuning depend on the task. In general, the LMs perform well on the sentiment analysis and machine translation tasks but struggle with tasks requiring the Norwegian language knowledge, commonsense reasoning, truthfulness, and instruction following. We summarize our findings below w.r.t. performance aggregation methods, human performance, task category, the effect of instruction tuning, Norwegian language variety, and LLM-as-a-judge evaluation.

Agreement on LM Rankings The agreement rate5 between the average normalized score and Borda’s count for the top-3 LMs is 66%. This discrepancy is because Borda’s count penalizes Mistral-Nemo-12B for its low performance on Norwegian language knowledge tasks, ranking NorMistral-11B and AI-Sweden/Llama-3-8B as the top-2 models, while Viking-13B takes third place instead of Mistral-Nemo-12B. However, the performance aggregation methods fully agree on the bottom-5 LMs, which include Viking-7B, Mistral-7B, NorMistral-7B-scratch, NorBLOOM-7B-warm, and NB-GPT-6B.

LMS vs. Human Baselines Comparing the LMs with our human baselines in Table 9 and Table 10 in Appendix E, we find that the LMs fall behind humans by 10% on Belebele, 14.4% on NorQuAD, 15.2% on NorOpenBookQA, 17.8% on NorCommonsenseQA, and 13.3% on NorTruthfulQA Multiple Choice. However, NorwAI-Llama2-7B slightly surpasses human performance on NCB by 1.2%. The results suggest that while LMs show promising in-context learning capabilities, there is still room for their improvement in world knowledge, truthfulness, and reading comprehension tasks.

Analysis on Task Categories We outline our key results based on the fine-grained results reported in Appendix E. No single LM consistently outperforms others across all task categories. The strongest performance is observed on the sentiment analysis tasks, with AI-Sweden/Llama-3-8B achieving the best score of 92.7 and its instruction-tuned version (NoReC Document) reaching 95.5.

<table>
  <tr>
    <th></th>
    <th>NB-GPT-6B</th>
    <th>GPT-SW3-6.7B</th>
    <th>NorwAI-Mistral-7B</th>
    <th>NorwAI-Llama2-7B</th>
    <th>NorBLOOM-7B-scratch</th>
    <th>NorMistral-7B-scratch</th>
    <th>Viking-7B</th>
    <th>NorMistral-11B</th>
    <th>Viking-13B</th>
    <th>NorMistral-7B-warm</th>
    <th>Mistral-7B</th>
    <th>AI-Sweden/Llama-3-8B</th>
    <th>Mistral-Nemo-12B</th>
    <th>Mistral-Nemo-12B-IT</th>
    <th>Mistral-7B-IT</th>
    <th>AI-Sweden/Llama-3-8B-IT</th>
    <th>Mistral-Nemo-12B-IT</th>
    <th>Mistral-Nemo-12B-IT</th>
  </tr>
  <tr>
    <td>NRK-Quiz-QA (ACC)</td>
    <td>-6.6</td>
    <td>-2.8</td>
    <td>-10</td>
    <td>-12</td>
    <td>-8.8</td>
    <td>-6.9</td>
    <td>-8.2</td>
    <td>-3.8</td>
    <td>-7.9</td>
    <td>-2.6</td>
    <td>-3.7</td>
    <td>-2.6</td>
    <td>-0.2</td>
    <td>-6.4</td>
    <td>-0.1</td>
    <td>-4.2</td>
    <td>-2.1</td>
  </tr>
  <tr>
    <td>NORBOOK (ACC)</td>
    <td>11</td>
    <td>5.3</td>
    <td>9.3</td>
    <td>8.1</td>
    <td>3.3</td>
    <td>14</td>
    <td>3.6</td>
    <td>8.9</td>
    <td>8</td>
    <td>4.2</td>
    <td>4.6</td>
    <td>4.6</td>
    <td>1.3</td>
    <td>3.7</td>
    <td>5.9</td>
    <td>1.9</td>
    <td>7.3</td>
  </tr>
  <tr>
    <td>NCSQA (ACC)</td>
    <td>13</td>
    <td>14</td>
    <td>11</td>
    <td>10</td>
    <td>11</td>
    <td>11</td>
    <td>9.4</td>
    <td>9.5</td>
    <td>8.1</td>
    <td>4.2</td>
    <td>4.6</td>
    <td>4.6</td>
    <td>1.3</td>
    <td>6.6</td>
    <td>2.0</td>
    <td>1.4</td>
    <td>0.8</td>
  </tr>
  <tr>
    <td>NTRQA (ACC)</td>
    <td>-0.5</td>
    <td>3</td>
    <td>8.5</td>
    <td>1.1</td>
    <td>1.5</td>
    <td>8.4</td>
    <td>6.4</td>
    <td>9.4</td>
    <td>4.6</td>
    <td>0.9</td>
    <td>0.9</td>
    <td>2.6</td>
    <td>2.6</td>
    <td>0.8</td>
    <td>6.6</td>
    <td>2.4</td>
    <td>5.2</td>
  </tr>
  <tr>
    <td>NTRQA (MC-ACC)</td>
    <td>-1.7</td>
    <td>-10</td>
    <td>-6.2</td>
    <td>-3.1</td>
    <td>-1.7</td>
    <td>-5</td>
    <td>-17</td>
    <td>-14</td>
    <td>-10</td>
    <td>-2.2</td>
    <td>-2.2</td>
    <td>-22</td>
    <td>-10</td>
    <td>-7</td>
    <td>-2.0</td>
    <td>-2.5</td>
    <td>-2.5</td>
  </tr>
</table>

Figure 2: Comparison of Bokmål and Nynorsk. Heatmap that shows the performance δ-scores between BM and NN on our multiple-choice QA and sentence completion tasks. NOBQA=NorOpenBookQA; NCSQA=NorCommonsenseQA; NTRQA =NorTruthfulQA. Higher values mean higher performance in BM.

On NorIdiom, GPT-SW3-6.7B delivers the best performance, followed by NorMistral-7B-warm. For NorCommonsenseQA, the performance of pre-trained LMs varies: BM scores range from 41.2 to 61, while NN scores range from 32.6 (Mistral-7B) to 51.6 (NorMistral-11B), suggesting limited in-context learning abilities for reasoning. The LMs also exhibit strong performance on Norwegian-specific quizzes (NRK-Quiz-QA) and tasks assessing elementary-level world knowledge (NorOpenBookQA), with the best-performing LMs including NorMistral-11B, AI-Sweden/Llama-3-8B, Mistral-7B, and Mistral-Nemo-12B. However, the LMs tend to generate less truthful answers in the open-ended QA setup (NorTruthfulQA Generation) compared to the multiple-choice setup (NorTruthfulQA Multiple Choice), highlighting potential challenges of evaluating open-ended QA in Norwegian.

Comparing Bokmål and Nynorsk We compute the performance δ-scores on multiple-choice and sentence completion tasks with parallel BM and NN datasets to compare LMs w.r.t. the Norwegian language variety. Figure 2 shows that the LMs generally perform better on BM on NorOpenBookQA, NorCommonsenseQA, and NorTruthfulQA Multiple Choice as opposed to NRK-Quiz-QA and NorIdiom. Instruction-tuning results in lower δ-scores on NRK-Quiz-QA and NorOpen-

5The proportion of top-k and bottom-k LMs that are consistently ranked by both performance aggregation methods.
<table>
  <tr>
    <th>Task</th>
    <th>\( k \)-shot</th>
    <th>Language switching</th>
    <th>Generation issues</th>
    <th>Input copying</th>
    <th>Redundant response</th>
    <th>Instruction misunderstanding</th>
    <th>Incorrect response</th>
  </tr>
  <tr>
    <td>NorIdiom (BM)</td>
    <td>0-shot</td>
    <td>40%</td>
    <td>0%</td>
    <td>8%</td>
    <td>20%</td>
    <td>28%</td>
    <td>4%</td>
  </tr>
  <tr>
    <td>ASK-GEC</td>
    <td>16-shot</td>
    <td>20%</td>
    <td>60%</td>
    <td>8%</td>
    <td>0%</td>
    <td>0%</td>
    <td>0%</td>
  </tr>
  <tr>
    <td>Tatoeba (En → BM)</td>
    <td>16-shot</td>
    <td>20%</td>
    <td>0%</td>
    <td>0%</td>
    <td>40%</td>
    <td>0%</td>
    <td>20%</td>
  </tr>
  <tr>
    <td>Tatoeba (En → NN)</td>
    <td>16-shot</td>
    <td>12%</td>
    <td>12%</td>
    <td>0%</td>
    <td>44%</td>
    <td>0%</td>
    <td>28%</td>
  </tr>
  <tr>
    <td>Overall</td>
    <td></td>
    <td>23%</td>
    <td>18%</td>
    <td>4%</td>
    <td>26%</td>
    <td>7%</td>
    <td>13%</td>
  </tr>
</table>

Table 5: Quantitative analysis results of the instruction-tuned LMs’ predictions on Norwegian language knowledge and machine translation tasks. Remaining 9% of the responses are manually classified as correct.

BookQA but leads to random guessing performance on NorIdiom for both BM and NN.

Effect of Instruction-tuning Instruction-tuning is one of the least explored research directions for Norwegian. Our results align with Wang et al. (2023); Bukharin et al. (2024) and show that instruction-tuning can yield both positive and negative effects depending on the task. For instance, instruction-tuning consistently improves the performance of Mistral-Nemo-12B and Meta/Llama-3-8B across most task categories, with the most notable improvements observed in multiple-choice QA and sequence-to-sequence generation tasks. At the same time, it can degrade the performance on tasks requiring Norwegian language knowledge and involve translating from English into BM and NN (see Table 8 and Table 11 in Appendix E).

Analysis of Unexpected Results To better understand the negative effects of instruction-tuning, we manually analyze the outputs of 100 instruction-tuned LMs on four tasks: NorIdiom (BM), ASK-GEC, and Tatoeba (En → BM/NN). The outputs are stratified by model and task, with 25 examples per task. The main quantitative results are presented in Table 5. Our analysis reveals that the instruction-tuned LMs frequently respond in English, Swedish, Danish, or a mix of these languages, even when the task requires output in BM or NN. The models also tend to produce redundant responses, often including assistant phrases such as How else can I help you?”. Other common issues include copying parts of the input, hallucinations, and repetitive content. In 13% of the cases, the LM understand the task but fail to produce a correct answer. We also observe that the LMs fail to interpret the NorIdiom task, leading to low scores. The results suggest that the model behavior can be improved by refining prompt design, tuning the inference hyperparameters, and explicitly specifying the target written standard in the instructions.

LLM-as-a-judge We report the LMs’ win-rates in Table 6. We find that NorMistral-7B-warm-IT and Mistral-Nemo-12B-IT consistently perform best across all LMs, while responses from AI-Sweden/Llama-3-8B-IT and Mistral-7B-IT are least preferred. NorMistral-7B-warm-IT achieves the highest win-rate on NorSummarize-Instruct, while there is a minor difference between the top-2 LMs on NorRewrite-Instruct. We analyze the agreement with humans, and self-preference, position, and language biases in Appendix F. Overall, meta-llama/Llama-3-3.3-70B-Instruct achieves a moderate agreement with humans (50% with ties and 66.7% without ties) and behaves fairly as a judge. We also find that there is an insignificant effect of the response position on the judge verdicts, and the instruction-tuned LMs often switch to English, Swedish, or Danish. The latter is supported by our manual analysis of the outputs (see Table 5).

6 Conclusion and Future Work

This work introduces NorEval, the largest benchmark for assessing the LM’s Norwegian language
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="5">NorRewrite-Instruct</th>
    <th colspan="5">NorSummarize-Instruct</th>
  </tr>
  <tr>
    <th>NotMistral-7B-warm-IT</th>
    <th>Mistral-Nemo-12B-IT</th>
    <th>Mistral-7B-IT</th>
    <th>Meta/Llama-3-8B-IT</th>
    <th>Average</th>
    <th>NotMistral-7B-warm-IT</th>
    <th>Mistral-Nemo-12B-IT</th>
    <th>Mistral-7B-IT</th>
    <th>Meta/Llama-3-8B-IT</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>NorMistral-7B-warm-IT</td>
    <td>—</td>
    <td>45.6</td>
    <td>92.2</td>
    <td>76.2</td>
    <td>99.5</td>
    <td>78.4</td>
    <td>—</td>
    <td>57.6</td>
    <td>92.5</td>
    <td>66.5</td>
    <td>99.5</td>
    <td>79.0</td>
  </tr>
  <tr>
    <td>Mistral-Nemo-12B-IT</td>
    <td>54.4</td>
    <td>—</td>
    <td>89.8</td>
    <td>80.6</td>
    <td>93.1</td>
    <td>79.5</td>
    <td>42.4</td>
    <td>—</td>
    <td>81.8</td>
    <td>62.1</td>
    <td>87.3</td>
    <td>68.4</td>
  </tr>
  <tr>
    <td>Mistral-7B-IT</td>
    <td>7.8</td>
    <td>10.2</td>
    <td>—</td>
    <td>47.4</td>
    <td>67.5</td>
    <td>33.2</td>
    <td>7.5</td>
    <td>18.2</td>
    <td>—</td>
    <td>36.9</td>
    <td>66.9</td>
    <td>32.4</td>
  </tr>
  <tr>
    <td>Meta/Llama-3-8B-IT</td>
    <td>23.8</td>
    <td>19.4</td>
    <td>52.6</td>
    <td>—</td>
    <td>64.7</td>
    <td>40.1</td>
    <td>33.5</td>
    <td>37.9</td>
    <td>63.1</td>
    <td>—</td>
    <td>71.4</td>
    <td>51.5</td>
  </tr>
  <tr>
    <td>AI-Sweden/Llama-3-8B-IT</td>
    <td>0.5</td>
    <td>6.9</td>
    <td>32.5</td>
    <td>35.3</td>
    <td>—</td>
    <td>18.8</td>
    <td>0.5</td>
    <td>12.7</td>
    <td>33.1</td>
    <td>28.6</td>
    <td>—</td>
    <td>18.7</td>
  </tr>
</table>

Table 6: Pair-wise expected win-rates (%) of the instruction-finetuned LMs on our instruction-following tasks. Cold-colored cells indicate high win-rate, while warm-colored cells indicate low win-rate.

understanding and generation capabilities on 24 human-created datasets. NorEval focuses on both Norwegian language varieties and spans nine task categories, ranging from Norwegian-specific & world knowledge to instruction following. We benchmark 19 open-source Norwegian generative LMs against each other and our established human baselines, analyzing their performance in various scenarios. Additionally, we present one of the first extensive evaluations of open Norwegian instruction-tuned LMs and their base counterparts in \( k \)-shot regimes, as well as via the LLM-as-a-judge approach. Our key findings indicate that the LMs struggle with tasks requiring Norwegian language knowledge, commonsense reasoning, truthfulness, and instruction following. The LMs generally perform better on BM compared to NN. Notably, instruction-tuning yields both positive and negative effects on the LM performance.

Our future work includes: (i) a more detailed evaluation of instruction-tuned LMs and instruction-tuning data mixtures; (ii) integration of novel datasets; (iii) establishment of human baselines on additional tasks; (iv) integration of test data decontamination methods. We hope that our benchmark and evaluation framework will facilitate more comprehensive comparisons of LMs within the context of Mainland Scandinavian languages and inspire collaborative efforts among NLP researchers and developers to advance reliable LMs and evaluation resources for Norwegian.

7 Limitations

While we present extensive empirical evaluations of a broad range of Norwegian LMs, we acknowledge the following limitations of our work.

Sampling Demonstrations. In the one- and 16-shot evaluation scenarios, demonstration examples are randomly sampled, which can facilitate label bias in our text classification and multiple-choice QA tasks (Zhao et al., 2021).

Multi-task Performance Aggregation. Aggregating evaluation results in multi-task benchmarking remains a challenging problem. We employ a combination of performance aggregation methods to mitigate the shortcomings of standard arithmetic mean aggregation: (i) score normalization to account for random baseline performance, and (ii) Borda’s count to address the heterogeneity of performance metrics. However, these methods have inherent limitations. In particular, we still need to average heterogeneous task-specific normalized performance scores to compute an overall score. Although Borda’s count relies on model rankings instead of performance scores, introducing a new LM can influence the final ranking due to the well-studies axiom of the independence of irrelevant alternatives (Arrow, 2012; Dougherty and Heckelman, 2020). Additionally, Borda’s count can treat several LMs as equivalent (or ties), which is not an empirical observation in our experiments.
Potential In-domain Evaluation  Our work does not account for potential in-domain evaluation of the instruction-tuned LMs, which can be instruction-tuned on similar tasks in English and other languages, potentially inflating their downstream performance.

LLM-as-a-judge  Automatic side-by-side evaluation using the LLM-as-a-judge approach is a well-established, complementary evaluation scenario that has demonstrated its efficiency for high-resource languages. However, its performance in low-resource languages remains unclear. We acknowledge that the reliability of our LLM-as-a-judge evaluation results requires further empirical validation. In particular, our analysis of agreement between the judge model and human annotators relies on the only available human preference dataset, which consists of 48 pairwise comparisons (see Appendix F). The limited sample size may affect the generalizability of our findings. Furthermore, our analysis focuses specifically on self-preference, language, and position biases; investigating other potential evaluation directions of judge models remains outside the scope of this work.

Human Baselines  We find that the language models slightly surpass the human performance on NCB (see §5). However, the results do not suggest that the models possess human-level capabilities in distinguishing between in- and correctly punctuated sentences, and evaluating them across more domains and example lengths is necessary to perform a more fine-grained performance analysis. While our annotators reach a strong inter-annotator agreement (see §4), we establish our human baselines on the test subsets of 50 examples only for BM. We acknowledge that increasing the number of examples could affect both the scores and agreement rates. Conducting human performance evaluation for NN could allow us to draw more conclusions regarding the human and model performance across both official written standards. This has not been done due to our limited resources, and we hope to address this in our future work.

Data Contamination  The increasing volume of open textual data can lead to unintended test data leakage in an LM’s pretraining corpus (e.g., Brown et al., 2020; Dubey et al., 2024; Zhang et al., 2024), which can promote the saturation of NLP benchmarks. We recognize the importance of this evaluation aspect and acknowledge that LM performance on NorEval datasets created from open text sources can be inflated. We encourage adherence to responsible LM development practices and recommend conducting test contamination analysis when benchmarking an LM on NorEval. Integrating unsupervised pretraining data detection methods into NorEval is left as a direction for our future work.

Evaluation Framework  NorEval is integrated into LM Harness Evaluation, a widely recognized open-source collaborative project that is subject to continuous improvements and advancements, which potentially affect its long-term compatibility, reproducibility, and usability.

Ethics Statement

Human Annotation  The hourly pay rate in our annotation projects (§3.2 and Appendix B.3) is regulated by the state and corresponds to the education level. The annotators’ submissions are stored anonymously. The annotators are warned about potentially sensitive topics in the dataset examples.

Inference Costs  Evaluating an LM on NorEval does not require any finetuning. The inference costs can be minimized with the help of distributed inference libraries supported by LM Evaluation Harness, such as Accelerate (Gugger et al., 2022) and vLLM (Kwon et al., 2023).

Potential Misuse  We acknowledge that NorEval can leak into and partially overlap with an LM’s pretraining corpus. We release NorEval for research and development purposes and encourage its responsible use.

Transparency & License  We release NorEval adhering to standard open-source research practices. The dataset licensing terms are provided in Table 7 (see Appendix A). Our codebase is available under the MIT license. Our comprehensive documentation and full annotation guidelines are available in our GitHub repository.

Use of AI-assistants  We use Grammarly6 to correct grammar, spelling, and phrasing errors in the text of this paper.

Acknowledgments

NorEval has developed from Mímir, a project on evaluating the impact of copyrighted data on pretraining Norwegian LMs (Rosa et al., 2025). We

6grammarly.com
thank our student annotators for their annotation efforts. We also thank our volunteers for their time and contribution to establishing our human baselines: Helene Bøsei Olsen, Lilja Charlotte Storset, Sondre Wold, Petter Mæhlum, Victoria Ovedie Chruickshank Langø, Egil Rønningstad, Emil Poiesz, Thea Tollersrud, and Asbjørn Sæther.

References

Kenneth J Arrow. 2012. Social choice and individual values, volume 12. Yale university press.

Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Fries, Maged Alshaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir Radev, Mike Tian-jian Jiang, and Alexander Rush. 2022. Prompt-Source: An integrated development environment and repository for natural language prompts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 93–104, Dublin, Ireland. Association for Computational Linguistics.

Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. The belebele benchmark: a parallel reading comprehension dataset in 122 language variants. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 749–775, Bangkok, Thailand. Association for Computational Linguistics.

Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, et al. 2024. Lessons from the trenches on reproducible evaluation of language models. arXiv preprint arXiv:2405.14782.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. Preprint, arXiv:2005.14165.

Alexander Bukharin, Shiyang Li, Zhengyang Wang, Jingfeng Yang, Bing Yin, Xian Li, Chao Zhang, Tuo Zhao, and Haoming Jiang. 2024. Data diversity matters for robust instruction tuning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3411–3425, Miami, Florida, USA. Association for Computational Linguistics.

Laurie Burchell, Ona de Gibert, Nikolay Arefyev, Mikko Aulamo, Marta Bañón, Mariia Fedorova, Liane Guillou, Barry Haddow, Jan Hajíč, Erik Henriksson, et al. 2025. An Expanded Massive Multilingual Dataset for High-Performance Language Technologies. arXiv preprint arXiv:2503.10267.

Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024. Humans or LLMs as the judge? a study on judgement bias. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8301–8327, Miami, Florida, USA. Association for Computational Linguistics.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv preprint arXiv:1803.05457.

Pierre Colombo, Nathan Noiry, Ekhine Irurzoki, and Stéphan Clémençon. 2022. What are The Best Systems? New Perspectives on NLP Benchmarking. Advances in Neural Information Processing Systems, 35:26915–26932.

Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world’s first truly open instruction-tuned llm.

Ona de Gibert, Graeme Nail, Nikolay Arefyev, Marta Bañón, Jelmer van der Linde, Shaoxiong Ji, Jaume Zaragoza-Bernabeu, Mikko Aulamo, Gema Ramírez-Sánchez, Andrey Kutuzov, Sampi Pyysalo, Stephan Oepen, and Jörg Tiedemann. 2024. A new massive multilingual dataset for high-performance language technologies. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 1116–1128, Torino, Italia. ELRA and ICCL.

Keith L Dougherty and Jac C Heckelman. 2020. The probability of violating arrow’s conditions. European Journal of Political Economy, 65:101936.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. 2023. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36:30039–30069.
Kenneth Enevoldsen, Márton Kardos, Niklas Muenninghoff, and Kristoffer L Nielbo. 2024. The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embedding. Advances in Neural Information Processing Systems, 37:40336–40358.

Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. 2024. Open LLM Leaderboard v2. Hugging Face.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noa’ch, Haonan Li, Kyle McDonell, Niklas Muenninghoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. A framework for few-shot language model evaluation.

Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Marc Sun, and Benjamin Bossan. 2022. Accelerate: Training and inference at scale made simple, efficient and adaptable. https://github.com/huggingface/accelerate.

Michael A. Hedderich, Lukas Lange, Heike Adel, Jan-nik Strötgen, and Dietrich Klakow. 2021. A survey on recent approaches for natural language processing in low-resource scenarios. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2545–2568, Online. Association for Computational Linguistics.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations.

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Adam Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. GPT-4o System Card. arXiv preprint arXiv:2410.21276.

Sardana Ivanova, Fredrik Andreassen, Matias Jentoft, Sondre Wold, and Lilja Øvrelid. 2023. NorQuAD: Norwegian question answering dataset. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 159–168, Tórshavn, Faroe Islands. University of Tartu Library.

Matias Jentoft. 2023. Grammatical Error Correction with Byte-level Language Models. Master’s thesis, University of Oslo.

Matias Jentoft and David Samuel. 2023. NoCoLA: The Norwegian corpus of linguistic acceptability. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 610–617, Tórshavn, Faroe Islands. University of Tartu Library.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825.

Fredrik Jørgensen, Tobias Aasmoe, Anne-Stine Ruud Husevåg, Lilja Øvrelid, and Erik Velldal. 2020. NorNE: Annotating named entities for Norwegian. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4547–4556, Marseille, France. European Language Resources Association.

Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Computational Linguistics.

Amir Hossein Kargaran, Ayyoob Imani, François Yvon, and Hinrich Schütze. 2023. GlotLID: Language identification for low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6155–6218, Singapore. Association for Computational Linguistics.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.

Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, and Huan Liu. 2025. From generation to judgment: Opportunities and challenges of llm-as-a-judge. Preprint, arXiv:2411.16594.

Peng Liu, Lemei Zhang, Terje Farup, Even W. Lauvrak, Jon Espen Igvaldsen, Simen Eide, Jon Atle Gulla, and Zhirong Yang. 2024. NLEBench+NorGLM: A comprehensive empirical analysis and benchmark dataset for generative language models in Norwegian. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5543–5560, Miami, Florida, USA. Association for Computational Linguistics.

Xinxi Lyu, Yizhong Wang, Hannaneh Hajishirzi, and Pradeep Dasigi. 2024. HREF: Human Response-Guided Evaluation of Instruction Following in Language Models. arXiv preprint arXiv:2412.15524.

Vladislav Mikhailov, Petter Mæhlum, Victoria Ovedie Chruckshank Lango, Erik Velldal, and Lilja Øvrelid. 2025. A collection of question answering datasets for Norwegian. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human
Language Technologies (NoDaLiDa/Baltic-HLT 2025), pages 397–407, Tallinn, Estonia. University of Tartu Library.

Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics.

Jørgen Johnsen Navjord and Jon-Mikkel Ryen Korsvik. 2023. Beyond extractive: advancing abstractive automatic text summarization in norwegian with transformers. Master’s thesis, Norwegian University of Life Sciences, Ås.

Dan Nielsen. 2023. ScandEval: A benchmark for Scandinavian natural language processing. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 185–201, Tórshavn, Faroe Islands. University of Tartu Library.

Qiang Ning, Zhili Feng, Hao Wu, and Dan Roth. 2018. Joint reasoning for temporal and causal relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2278–2288, Melbourne, Australia. Association for Computational Linguistics.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.

Lilja Øvrelid and Petter Hohle. 2016. Universal Dependencies for Norwegian. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1579–1585, Portorož, Slovenia. European Language Resources Association (ELRA).

Lilja Øvrelid, Petter Mehlum, Jeremy Barnes, and Erik Velldal. 2020. A fine-grained sentiment dataset for Norwegian. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5025–5033, Marseille, France. European Language Resources Association.

Arjun Panickssery, Samuel Bowman, and Shi Feng. 2024. Llm evaluators recognize and favor their own generations. Advances in Neural Information Processing Systems, 37:68772–68802.

Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Martin Jaggi, Leandro von Werra, and Thomas Wolf. 2024. FineWeb2: A sparkling update with 1000s of languages.

Nazneen Rajani, Lewis Tunstall, Edward Beeching, Nathan Lambert, Alexander M. Rush, and Thomas Wolf. 2023. No robots. https://huggingface.co/datasets/HuggingFaceH4/no_robots.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics.

Mark Rofin, Vladislav Mikhailov, Mikhail Florinsky, Andrey Kravchenko, Tatiana Shavrina, Elena Tubulabina, Daniel Karabekyan, and Ekaterina Artemova. 2023. Vote’n’rank: Revision of benchmarking with social choice theory. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 670–686, Dubrovnik, Croatia. Association for Computational Linguistics.

Javier de la Rosa, Vladislav Mikhailov, Lemei Zhang, Freddy Wetjen, David Samuel, Peng Liu, Rolv-Arild Braaten, Petter Mehlum, Magnus Breder Birkenes, Andrey Kutuzov, Tita Enstad, Hans Christian Færsethås, Svein Arne Brygfjeld, Jon Atle Gulla, Stephan Oepen, Erik Velldal, Wilfred Østgulen, Lilja Øvrelid, and Aslak Sira Myhre. 2025. The impact of copyrighted material on large language models: A Norwegian perspective. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025), pages 544–560, Tallinn, Estonia. University of Tartu Library.

Sebastian Ruder. 2021. Challenges and Opportunities in NLP Benchmarking.

Ananya B. Sai, Akash Kumar Mohankumar, and Mitesh M. Khapra. 2022. A survey of evaluation metrics used for nlg systems. ACM Comput. Surv., 55(2).

David Samuel, Andrey Kutuzov, Samia Touileb, Erik Velldal, Lilja Øvrelid, Egil Ronningstad, Elina Sigdel, and Anna Palatkina. 2023. NorBench – a benchmark for Norwegian language models. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 618–633, Tórshavn, Faroe Islands. University of Tartu Library.

David Samuel, Vladislav Mikhailov, Erik Velldal, Lilja Øvrelid, Lucas Georges Gabriel Charpentier, Andrey Kutuzov, and Stephan Oepen. 2025. Small languages, big models: A study of continual training on languages of Norway. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025), pages 573–608, Tallinn, Estonia. University of Tartu Library.

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2023. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research.
Kari Tenfjord, Paul Meurer, and Knut Hofland. 2006. The ASK corpus - a language learner corpus of Norwegian as a second language. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), Genoa, Italy. European Language Resources Association (ELRA).

Jörg Tiedemann. 2020. The tatoeba translation challenge – realistic data sets for low resource and multilingual MT. In Proceedings of the Fifth Conference on Machine Translation, pages 1174–1182, Online. Association for Computational Linguistics.

Samia Touileb, Vladislav Mikhailov, Marie Ingeborg Kroka, Lilja Øvrelied, and Erik Velldal. 2025. Benchmarking abstractive summarisation: A dataset of human-authored summaries of Norwegian news articles. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025), pages 729–738, Tallinn, Estonia. University of Tartu Library.

Erik Velldal, Lilja Øvrelied, Eivind Alexander Bergem, Cathrine Stadsnes, Samia Touileb, and Fredrik Jorgensen. 2018. NoReC: The Norwegian review corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).

Erik Velldal, Lilja Øvrelied, and Petter Hohle. 2017. Joint UD parsing of Norwegian Bokmål and Nynorsk. In Proceedings of the 21st Nordic Conference on Computational Linguistics, pages 1–10, Gothenburg, Sweden. Association for Computational Linguistics.

Anton Voronov, Lena Wolf, and Max Ryabinin. 2024. Mind your format: Towards consistent evaluation of in-context learning improvements. In Findings of the Association for Computational Linguistics: ACL 2024, pages 6287–6310, Bangkok, Thailand. Association for Computational Linguistics.

Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. 2024. Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9440–9450, Bangkok, Thailand. Association for Computational Linguistics.

Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. 2023. How far can camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information Processing Systems, 36:74764–74786.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics.

Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Yang, and Hai Li. 2024. Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models. arXiv preprint arXiv:2404.02936.

Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12697–12706. PMLR.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
A  NorEval: Dataset Descriptions, Examples, and Prompts

<table>
  <tr>
    <th>Dataset</th>
    <th>Language</th>
    <th># Train</th>
    <th># Test</th>
    <th># Prompts</th>
    <th>Method</th>
    <th>Task Type</th>
    <th>Task Category</th>
    <th>Performance Metrics</th>
    <th>License</th>
  </tr>
  <tr>
    <th colspan="10">Peer-reviewed Norwegian datasets</th>
  </tr>
  <tr>
    <td>NorReC Sentence</td>
    <td>BM</td>
    <td>3.89k</td>
    <td>583</td>
    <td>5</td>
    <td>Human-created</td>
    <td>Text classification</td>
    <td>Sentiment analysis</td>
    <td>F1<sub>r</sub>, Accuracy score</td>
    <td>CC BY-NC 4.0</td>
  </tr>
  <tr>
    <td>NorReC Document</td>
    <td>BM</td>
    <td>23.4k</td>
    <td>2.9k</td>
    <td>5</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>NorQuAD</td>
    <td>BM</td>
    <td>3.81k</td>
    <td>472</td>
    <td>5</td>
    <td>Human-created</td>
    <td>Generative QA</td>
    <td>Reading Comprehension</td>
    <td>F1, Exact match</td>
    <td>CC0-1.0</td>
  </tr>
  <tr>
    <td>ASK-GEC</td>
    <td>BM</td>
    <td>36.4k</td>
    <td>4.75k</td>
    <td>5</td>
    <td>Human-created</td>
    <td>Seq2seq generation</td>
    <td>Norwegian language knowledge</td>
    <td>ERRANT</td>
    <td>CC BY 4.0</td>
  </tr>
  <tr>
    <td>Betebele</td>
    <td>BM</td>
    <td>x</td>
    <td>900</td>
    <td>5</td>
    <td>Human-translated</td>
    <td>Multiple-choice QA</td>
    <td>Reading Comprehension</td>
    <td>Accuracy score</td>
    <td>CC BY-SA 4.0</td>
  </tr>
  <tr>
    <td>Tatoeba</td>
    <td>En ↔ BM</td>
    <td>5.2k</td>
    <td>4.5k</td>
    <td>8</td>
    <td>Human-created</td>
    <td>Seq2seq generation</td>
    <td>Machine translation</td>
    <td>BLEU, BERTScore</td>
    <td>CC-BY-2.0</td>
  </tr>
  <tr>
    <td>En ↔ NN</td>
    <td>504</td>
    <td>459</td>
    <td>8</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>NorOpenBookQA</td>
    <td>BM</td>
    <td>2.8k</td>
    <td>163</td>
    <td>5</td>
    <td>Human-created & human-translated</td>
    <td>Multiple-choice QA</td>
    <td>Norwegian-specific & world knowledge</td>
    <td>Accuracy score</td>
    <td>MIT</td>
  </tr>
  <tr>
    <td>NN</td>
    <td>376</td>
    <td>90</td>
    <td>5</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>NRK-Quiz-QA</td>
    <td>BM</td>
    <td>x</td>
    <td>3.6k</td>
    <td>5</td>
    <td>Human-created</td>
    <td>Multiple-choice QA</td>
    <td>Norwegian-specific & world knowledge</td>
    <td>Accuracy score</td>
    <td>MIT</td>
  </tr>
  <tr>
    <td>NN</td>
    <td>x</td>
    <td>1.3k</td>
    <td>5</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>NorCommonsenseQA</td>
    <td>BM</td>
    <td>x</td>
    <td>693</td>
    <td>5</td>
    <td>Human-created & human-translated</td>
    <td>Multiple-choice QA</td>
    <td>Commonsense reasoning</td>
    <td>Accuracy score</td>
    <td>MIT</td>
  </tr>
  <tr>
    <td>NN</td>
    <td>x</td>
    <td>95</td>
    <td>5</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>NorTruthfulQA MC</td>
    <td>BM</td>
    <td>x</td>
    <td>488</td>
    <td>5</td>
    <td>Human-created & human-translated</td>
    <td>Multiple-choice QA</td>
    <td>Truthfulness</td>
    <td>Accuracy score</td>
    <td>MIT</td>
  </tr>
  <tr>
    <td>NN</td>
    <td>x</td>
    <td>57</td>
    <td>5</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>NorTruthfulQA Gen</td>
    <td>BM</td>
    <td>x</td>
    <td>346</td>
    <td>5</td>
    <td>Human-created & human-translated</td>
    <td>Generative QA</td>
    <td>Truthfulness</td>
    <td>BLEU, ROUGE-L</td>
    <td>MIT</td>
  </tr>
  <tr>
    <td>NN</td>
    <td>x</td>
    <td>125</td>
    <td>5</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>NorSumm</td>
    <td>BM</td>
    <td>30</td>
    <td>33</td>
    <td>6</td>
    <td>Human-created</td>
    <td>Seq2seq generation</td>
    <td>Text summarization</td>
    <td>ROUGE-L, BERTScore</td>
    <td>CC0-1.0</td>
  </tr>
  <tr>
    <td>NN</td>
    <td>30</td>
    <td>33</td>
    <td>6</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <th colspan="10">Novel datasets for Norwegian (ours)</th>
  </tr>
  <tr>
    <td>NorRewrite-Instruct</td>
    <td>BM</td>
    <td>x</td>
    <td>144</td>
    <td>144</td>
    <td>Human-created</td>
    <td>Seq2seq generation</td>
    <td>Instruction following</td>
    <td>chrF, BERTScore</td>
    <td>MIT</td>
  </tr>
  <tr>
    <td>NorSummarize-Instruct</td>
    <td>NN</td>
    <td>x</td>
    <td>197</td>
    <td>197</td>
    <td>Human-created</td>
    <td>Seq2seq generation</td>
    <td>Instruction following</td>
    <td>chrF, BERTScore</td>
    <td>MIT</td>
  </tr>
  <tr>
    <td>NorIdiom</td>
    <td>BM</td>
    <td>x</td>
    <td>3.4k</td>
    <td>5</td>
    <td>Human-created</td>
    <td>Sentence completion</td>
    <td>Norwegian language knowledge</td>
    <td>F1, Exact match</td>
    <td>CC0-1.0</td>
  </tr>
  <tr>
    <td>NN</td>
    <td>x</td>
    <td>89</td>
    <td>5</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>NCB</td>
    <td>BM</td>
    <td>x</td>
    <td>840</td>
    <td>x</td>
    <td>Human-created</td>
    <td>Sentence ranking</td>
    <td>Norwegian language knowledge</td>
    <td>Accuracy score</td>
    <td>CC BY-NC 4.0</td>
  </tr>
</table>

Table 7: Overview of the datasets in NorEval w.r.t. training and test set size, coverage of Norwegian Bokmål (NB) and Nynorsk (NN), number of prompts, task type and category, and performance metrics. En=English.

This appendix presents an overview of the 24 datasets included in NorEval (also see Table 7).

NCB
The Norwegian Comma Benchmark (NCB) is a collection of 840 human-written Norwegian sentence pairs. The sentences are manually collected from publicly available sources such as articles and governmental reports. The sentences aim to be representative of Norwegian non-fiction, in particular governmental prose. Each sentence pair tests one Norwegian comma rule: one sentence is correctly punctuated, while the other contains faulty comma usage.

• correct: “Spørsmålet om å begrense forvaltningens arbeidsbyrde ble viet stor oppmerksomhet.”
• wrong: “Spørsmålet om å begrense forvaltningens arbeidsbyrde, ble viet stor oppmerksomhet.”

Task Formulation  Given a pair of sentences, the task is to select a correctly punctuated sentence by ranking both sentences based on their probability. The performance metric is the accuracy score.
NorIdiom
NorIdiom is designed to evaluate an LM’s knowledge of 3.5k common Norwegian idioms and phrases. Each task example consists of the first \( N - 1 \) words of an idiom, and a list of accepted last words to complete the idiom.

• idiom_start: “bite på”
• accepted_completions: “kroken”, “agnet”

Task formulation  The task is to generate the last word of an incomplete idiom. We maximize the F1 and exact match performance scores over the list of accepted completions.

Prompt A (BM and NN):
    Fullfør dette uttrykket: {{idiom_start}}

Prompt B (BM):
    Skriv fortsettelsen av idiomet {{idiom_start}}

Prompt B (NN):
    Skriv fortsetjinga av idiomet {{idiom_start}}

Prompt C (BM):
    Hvordan fortsetter uttrykket "{{idiom_start}}"?

Prompt C (NN):
    Korleis fortset uttrykket "{{idiom_start}}"?

Prompt D (BM):
    Fullfør vendingen "{{idiom_start}}"

Prompt D (NN):
    Fullfør vendinga: {{idiom_start}}

Prompt E (BM and NN):
    {{idiom_start}}

Belebele
Belebele is a multiple-choice QA dataset spanning 122 language variants. Each question has four multiple-choice answers a short passage.

Task Formulation  The task is to select a correct answer option given a passage and a question. The performance metric is the accuracy score.

• passage: “Så og si nesten alle PC-er som benyttes i dag, baseres på manipulering av informasjon som er kodet med binære tall. Et binært tall kan kun ha én av to verdier, dvs. 0 eller 1. Disse tallene omtales som binersifre – eller biter, for å bruke datasjargon.”
• question: “Hvilke av følgende er et eksempel et binært tall med fem biter, ifølge avsnittet?”
• answer_1: 1010
• answer_2: 12001
• answer_3: 10010
• answer_4:110101
• correct_answer_num: 3
Prompt A:
1 Tekst: {{passage}}
2 Spørsmål: {{question}}
3 A: {{answer_1}}
4 B: {{answer_2}}
5 C: {{answer_3}}
6 D: {{answer_4}}
7 Svar: {prediction:A/B/C/D}

Prompt B:
1 Bakgrunn: {{passage}}
2 Spørsmål: {{question}}
3 Svaralternativer:
4 - {{answer_1}}
5 - {{answer_2}}
6 - {{answer_3}}
7 - {{answer_4}}
8 Svar: {prediction:{{answer_1}}/{{answer_2}}/{{answer_3}}/{{answer_4}}}

Prompt C:
1 {{question}}
2 Hvilket av følgende mulige svar er det riktige?
3 A: {{answer_1}}
4 B: {{answer_2}}
5 C: {{answer_3}}
6 D: {{answer_4}}
7 Svar: {prediction:A/B/C/D}

Prompt D:
1 Svar på følgende spørsmål: {{question}}
2 Svaret skal baseres på følgende tekst:
3 {{passage}}
4 Velg et svar fra denne listen:
5 - {{answer_1}}
6 - {{answer_2}}
7 - {{answer_3}}
8 - {{answer_4}}
9 Svar: {prediction:{{answer_1}}/{{answer_2}}/{{answer_3}}/{{answer_4}}}

Prompt E:
1 {{passage}}
2
3 {{question}}
4
5 A: {{answer_1}}
6 B: {{answer_2}}
7 C: {{answer_3}}
8 D: {{answer_4}}
9
10 Er det riktige svaret A, B, C, eller D? {prediction:A/B/C/D}
NorQuAD
NorQuAD consists of 4.7k manually created examples based on Wikipedia and news articles following the SQuAD design (Rajpurkar et al., 2016).

• title: “Ordspråk
• context: “Ordspråk eller ordtak er korte, velformulerte og poengterte setninger som på en konkret måte uttrykker livsvisdom, allmenngyldige sannheter, erfaringer, leveregler eller betraktninger av forskjellig slag. Ordspråk kan også inneholde forklaringer av naturfenomener, skikker og seder. Ordspråk har en fast ordlyd som er kjent og blir sitert, for eksempel for å kommentere noe eller for å gi et råd. Mange ordspråk har uklar opprinnelse og er en del av gammel folkediktning og en muntlig fortellertradisjon. Det er også mange som er sitater fra bøker og fortellinger med kjent opphav, for eksempel fra Bibelen og Håvamål, selv om begrepet ordspråk ofte brukes om folkelige uttrykk uten kjent forfatter. Ordspråk kan være internasjonale, nasjonale og regionale og finnes i et nærmest uendelig antall og i en mengde varianter over hele verden. Studiet av ordspråk kalles paromiologi. Også fraseologien beskriver etablerte flerordsenheter og -forbindelser i et språk, særlig faste uttrykk og idiomer, men også tekster som ordspråk.”
• question: “Hvordan er opprinnelsen til mange ordspråk?”
• answer: “uklar”

Task Formulation The task is to extract the answer from the context given a question. We formulate it as a sequence-to-sequence problem, where the LM receives the context and the question as the input and is expected to generate the answer. The performance metrics are exact match (the percentage of predictions that exactly match the gold answer) and F1-score (the average N-gram overlap between the prediction and the gold answer treated as bag-of-words).

Prompt A:
Tittel: {{title}}
Tekst: {{passage}}
Spørsmål: {{question}}
Svar: {{prediction}}

Prompt B:
Tittel: {{title}}
Tekst: {{passage}}
Gitt teksten over, hva er svaret på følgende spørsmål? "{{question}}"
Svar: {{prediction}}

Prompt C:
Tittel: {{title}}
Tekst: {{passage}}
Svar på følgende: {{question}}
Svar: {{prediction}}
Prompt D:
Tittel: {{title}}
Tekst: {{passage}}
Hvordan kan man svare på spørsmålet "{{question}}", gitt teksten over?
Svar:{{prediction}}

Prompt E:
Tittel: {{title}}
Tekst: {{passage}}
Gitt teksten over, besvar følgende spørsmål: "{{question}}"
Svar: {{prediction}}

NoReC Sentence
NoReC Sentence is a dataset for sentence-level sentiment analysis in Norwegian, derived from NoReC_fine (Øvrelid et al., 2020). The annotations have been aggregated at the sentence-level, by only keeping sentences that contain sentiment annotations of either positive or negative polarity.

Task Formulation  The task is framed as a binary classification problem. The LM is required to predict if a given review has a positive or negative sentiment. The target performance metric is the macro-average F1-score.

• review: “En mer allsidig og tilkoblingsvennlig skjerm har vi knapt sett .”
• sentiment: 1 (positive).

Prompt A:
Tekst: {{text}}
Sentiment: {prediction:positiv/negativ}

Prompt B:
{{text}}
Er denne setningen "positiv" eller "negativ"? {prediction:positiv/negativ}

Prompt C:
{{text}}
Hva slags sentiment uttrykker anmelderen? {prediction:positiv/negativ}

Prompt D:
{{text}}
Er anmeldelsen "positiv" eller "negativ"? {prediction:positiv/negativ}

Prompt E:
{{text}}
Er denne setningen positiv eller negativ? {prediction:positiv/negativ}

NoReC Document
NoReC Document is a dataset for document-level sentiment analysis derived from NoReC (Veldal et al., 2018) by keeping documents that have positive (ratings 5–6) or negative (ratings 1–3) sentiment.
Task Formulation  The task is framed as a binary classification problem. The LM is required to predict if a given review has a positive or negative sentiment. The target performance metric is the macro-average F1-score.

Prompt A:
1 Tekst: {{text}}
2 Sentiment: {prediction:positiv/negativ}

Prompt B:
1 Tekst: {{text}}
2 Er anmeldelsen "positiv" eller "negativ"? {prediction:positiv/negativ}

Prompt C:
1 Er polariteten til følgende anmeldelse positiv eller negativ?
2 Anmeldelse: {{text}}
3 Anmeldelsen er {prediction:positiv/negativ}

Prompt D:
1 Anmeldelse: {{text}}
2 Er anmelderen positiv eller negativ? {prediction:positiv/negativ}

Prompt E:
1 Anmeldelse: {{text}}
2 Vil du oppsummere anmeldelsen som "bra" eller "dårlig"? {prediction:bra/dårlig}

NorCommonsenseQA
NorCommonsenseQA is developed to assess the LM’s commonsense reasoning abilities. It includes 1.1k examples in BM and NN, each comprising a question and five answer choices.

• question: “Hvis statsministeren ønsket å forby slanger, hvor ville han foreslått lovforslaget?”
• answer_1: “På gata”
• answer_2: “I en tropisk skog”
• answer_3: “I Edens hage”
• answer_4: “På Eidsvoll”
• answer_5: “I Stortinget” (correct)

Task Formulation  The task is to select a correct answer to given a question. The performance metric is the accuracy score.

Prompt A (BM and NN):
1 Spørsmål: {{question}}
2 Svar: {prediction:{{answer_1}}/{{answer_2}}/{{answer_3}}/{{answer_4}}/{{answer_5}}}

Prompt B (BM):
1 {{question}}
2 Hvilket av følgende mulige svar er det riktige?
3 A: {{answer_1}}
4 B: {{answer_2}}
5 C: {{answer_3}}
6 D: {{answer_4}}
7 E: {{answer_5}}
8 Svar: {prediction:A/B/C/D/E}
Prompt B (NN):
{{question}}
Kva av følgande moglege svar er det rette?
A: {{answer_1}}
B: {{answer_2}}
C: {{answer_3}}
D: {{answer_4}}
E: {{answer_5}}
Svar: {prediction:A/B/C/D/E}

Prompt C (BM):
Gitt alternativene under, hva er svaret på følgende spørsmål: {{question}}

Alternativer:
- {{answer_1}}
- {{answer_2}}
- {{answer_3}}
- {{answer_4}}
- {{answer_5}}

Svar: {prediction:{{answer_1}}/{{answer_2}}/{{answer_3}}/{{answer_4}}/{{answer_5}}}

Prompt C (NN):
Gitt alternativa under, kva er svaret på følgande spørsmål: {{question}}

Alternativ:
- {{answer_1}}
- {{answer_2}}
- {{answer_3}}
- {{answer_4}}
- {{answer_5}}

Svar: {prediction:A/B/C/D/E}

Prompt D (BM):
{{question}}
Velg riktig svar blant disse alternativene:
- {{answer_1}}
- {{answer_2}}
- {{answer_3}}
- {{answer_4}}
- {{answer_5}}

Svar: {prediction:{{answer_1}}/{{answer_2}}/{{answer_3}}/{{answer_4}}/{{answer_5}}}

Prompt D (NN):
{{question}}
Vel rett svar blant desse alternativa:
- {{answer_1}}
- {{answer_2}}
- {{answer_3}}
- {{answer_4}}
- {{answer_5}}

Svar: {prediction:{{answer_1}}/{{answer_2}}/{{answer_3}}/{{answer_4}}/{{answer_5}}}
Prompt E (BM):
1   {{question}}
2   A: {{answer_1}}
3   B: {{answer_2}}
4   C: {{answer_3}}
5   D: {{answer_4}}
6   E: {{answer_5}}
7
8   Er det riktige svaret A, B, C, D, eller E?
9
10  Svar: {prediction:A/B/C/D/E}

Prompt E (NN):
1   {{question}}
2   A: {{answer_1}}
3   B: {{answer_2}}
4   C: {{answer_3}}
5   D: {{answer_4}}
6   E: {{answer_5}}
7
8   Er det rette svaret A, B, C, D, eller E?
9
10  Svar: {prediction:A/B/C/D/E}

NRK-Quiz-QA

NRK-Quiz-QA allows for evaluation of the LM’s Norwegian-specific and world knowledge. NRK-Quiz-QA includes 4.9k examples in BM and NN from more than 500 quizzes covering various topics on the Norwegian language and culture. Each example contains a question and 2 to 5 answer choices.

• question: “Æ træng læsta: Læsta er kjekt å ha. I alle fall sånn innimellom. Men hva er det for noe?”
• answer_1: “Venner”
• answer_2: “Lesestoff”
• answer_3: “Ro”
• answer_4: “Ullsokker” (correct)

Task Formulation The task is to select a correct answer to given a question. The performance metric is the accuracy score.

Prompt A (BM and NN):
1   Spørsmål: {{question}}
2
3   Svar: {prediction:{{answer_1}}/{{answer_2}}/{{answer_3}}/{{answer_4}}}
Prompt B (BM):
{{question}}
Svaralternativer:
- {{answer_1}}
- {{answer_2}}
- {{answer_3}}
- {{answer_4}}
Hva er riktig svar?
Svar: {prediction:{{answer_1}}/{{answer_2}}/{{answer_3}}/{{answer_4}}}

Prompt B (NN):
{{question}}
{{question}}
Svaralternativer:
- {{answer_1}}
- {{answer_2}}
- {{answer_3}}
- {{answer_4}}
Kva er rett svar?
Svar: {prediction:{{answer_1}}/{{answer_2}}/{{answer_3}}/{{answer_4}}}

Prompt C (BM):
{{question}}
A: {{answer_1}}
B: {{answer_2}}
C: {{answer_3}}
D: {{answer_4}}
Er det riktige svaret A, B, C, eller D?
Svar: {prediction:A/B/C/D}

Prompt C (NN):
{{question}}
A: {{answer_1}}
B: {{answer_2}}
C: {{answer_3}}
D: {{answer_4}}
Er det rette svare A, B, C, eller D?
Svar: {prediction:A/B/C/D}

Prompt D (BM and NN):
Spørsmål: {{question}}
A: {{answer_1}}
B: {{answer_2}}
C: {{answer_3}}
D: {{answer_4}}
Svar: {prediction:A/B/C/D}
Prompt E (BM):
1   {{question}}
2   Velg riktig svar blant disse alternativene:
3   - {{answer_1}}
4   - {{answer_2}}
5   - {{answer_3}}
6   - {{answer_4}}
7
8   Svar: {prediction:{{answer_1}}/{{answer_2}}/{{answer_3}}/{{answer_4}}}

Prompt E (NN):
1   {{question}}
2   Vel rett svar blant desse alternativa:
3   - {{answer_1}}
4   - {{answer_2}}
5   - {{answer_3}}
6   - {{answer_4}}
7
8   Svar: {prediction:{{answer_1}}/{{answer_2}}/{{answer_3}}/{{answer_4}}}

NorOpenBookQA

NorOpenBookQA is designed to evaluate the LM’s world knowledge. NorOpenBookQA counts 3.5k examples in BM and NN, each consisting of an elementary-level science question, four answer choices, and a factual statement that presents the evidence necessary to determine the correct answer.

• question: “Hva er mykest?”
• answer_1: “Marshmallows”
• answer_1: “Stål”
• answer_1: “Diamant”
• answer_1: “Saltstenger”
• fact: “Et mineral som kan skrapes av en fingernegl regnes som mykt”

Task Formulation The task is to select a correct answer to given a question. The performance metric is the accuracy score.

Prompt A (BM and NN):
1   {{fact}}
2   {{question}} {prediction:{{answer_1}}/{{answer_2}}/{{answer_3}}/{{answer_4}}}

Prompt B (BM):
1   Faktatekst: {{fact}}
2   Spørsmål til teksten: {{question}}
3
4   Svaralternativer:
5   - {{answer_1}}
6   - {{answer_2}}
7   - {{answer_3}}
8   - {{answer_4}}
9
10  Hva er riktig svar? {prediction:{{answer_1}}/{{answer_2}}/{{answer_3}}/{{answer_4}}}
Prompt B (NN):
1  Faktatekst: {{fact}}
2  Spørsmål til teksten: {{question}}

4  Svaralternativer:
5   - {{answer_1}}
6   - {{answer_2}}
7   - {{answer_3}}
8   - {{answer_4}}

10 Kva er rett svar? {prediction:{{answer_1}}/{{answer_2}}/{{answer_3}}/{{answer_4}}}

Prompt C (BM):
1  {{fact}}
2  {{question}}
3  A: {{answer_1}}
4  B: {{answer_2}}
5  C: {{answer_3}}
6  D: {{answer_4}}

8  Er det riktige svaret A, B, C, eller D?

10 Svar: {prediction:A/B/C/D}

Prompt C (NN):
1  {{fact}}
2  {{question}}
3  A: {{answer_1}}
4  B: {{answer_2}}
5  C: {{answer_3}}
6  D: {{answer_4}}

8  Er det rette svare A, B, C, eller D?

10 Svar: {prediction:A/B/C/D}

Prompt D (BM and NN):
1  Bakgrunn: {{fact}}

3  Spørsmål: {{question}}
4  A: {{answer_1}}
5  B: {{answer_2}}
6  C: {{answer_3}}
7  D: {{answer_4}}

9  Svar: {prediction:A/B/C/D}

Prompt E (BM):
1  Ta utgangspunkt i følgende fakta når du svarer på spørsmålet: {{fact}}

3  {{question}}
4  Velg riktig svar blant disse alternativene:
5   - {{answer_1}}
6   - {{answer_2}}
7   - {{answer_3}}
8   - {{answer_4}}
Svar: {prediction:{{answer_1}}/{{answer_2}}/{{answer_3}}/{{answer_4}}}

Prompt E (NN):
Ta utgangspunkt i følgande fakta når du svarar på spørsmålet: {{fact}}
{{question}}
Vel rett svar blant desse alternativa:
- {{answer_1}}
- {{answer_2}}
- {{answer_3}}
- {{answer_4}}

Svar: {prediction:{{answer_1}}/{{answer_2}}/{{answer_3}}/{{answer_4}}}

NorSumm
NorSumm is an abstractive text summarization dataset of news articles taken from the news part of the text sources of the Norwegian UD Treebank. Each news article is summarized in several versions in both BM and NN.

Task Formulation The task is an abstractive text summarization, where the LM is required to summarize a given news article. We use a combination of standard performance metrics (ROUGE-Land BERTScore), and maximize each performance score over the list of human references.

Prompt A (BM):
Skriv en oppsummering av følgende artikkel med kun noen få punkter: {{article}}
Oppsummering: {{prediction}}

Prompt A (NN):
Skriv ei oppsummering av følgande artikkel med berre nokre få punkt: {{article}}
Oppsummering: {{prediction}}

Prompt B (BM):
Oppsummer følgende artikkel med noen få setninger: {{article}}
Oppsummering: {{prediction}}

Prompt B (NN):
Oppsummer følgande artikkel med nokre få setningar: {{article}}
Oppsummering: {{prediction}}

Prompt C (BM):
{{article}}
Skriv en kort og presis oppsummering av teksten over. <...> Oppsummeringen skal inneholde ↳ maksimalt 700 tegn, inkludert mellomrom. {{prediction}}

Prompt C (NN):
{{article}}
Skriv ein kort og presis oppsummering av teksten over. <...> Oppsummeringa skal innehalde ↳ maksimalt 700 tegn, inkludert mellomrom. {{prediction}}

Prompt D (BM):
Gi et kortfattet sammendrag av følgende tekst: {{article}} {{prediction}}
Prompt D (NN):
1   Gje eit kortfatta samandrag av følgande tekst: {{article}} {{prediction}}

Prompt E (BM):
1   Lag en kort oppsummering som sammenfatter den følgende teksten i noen få punkter:
2   {{article}}
3
4   Oppsummering: {{prediction}}

Prompt E (NN):
1   Lag ein kort oppsummering som samanfattar den følgande teksten i nokre få punkt:
2   {{article}}
3
4   Oppsummering: {{prediction}}

Prompt F (BM):
1   Hele artikkelen:
2   {{article}}
3
4   Hovedpunkter: {{prediction}}

Prompt F (NN):
1   Heile artikkelen:
2   {{article}}
3
4   Hovudpunkt: {{prediction}}

ASK-GEC

ASK-GEC is focused on the task of grammatical error correction and is derived from the Norsk Anderspråkskorpus (Tenfjord et al., 2006). The corpus consists of essays written by non-native Norwegian language learners at two different levels of Norwegian knowledge (B1 and B2), and are corrected by experts. Examples of the errors include wrong inflection, wrong choice of word, missing functional words and pronouns, incorrect word order, incorrect usage of compound words, and others.

Task Formulation  The task is to correct grammatical errors in the input. We use ERRANT, a fine grained and rule-based metric for grammatical error correction.

Prompt A:
1   Tekst: {{text}}
2   Korreksjon: {{prediction}}

Prompt B:
1   Tekst: {{text}}
2   Rettet versjon: {{prediction}}

Prompt C:
1   Skriv om følgende tekst slik at den blir grammatisk korrekt: {{text}}
2   Korreksjon: {{prediction}}

Prompt D:
1   Original versjon: {{text}}
2   Korrekturlest og rettet versjon: {{prediction}}
Prompt E:
1   Rett opp grammatiske feil i denne teksten: {{text}}
2   Korreksjon: {{prediction}}

Tatoeba
Tatoeba is a multilingual machine translation benchmark derived from user-contributed translations.

Task Formulation   The task is to generate a translation in a target language given a sentence in a source language. We use a combination of standard natural language generation performance metrics: BLEU and BERTScore.

English → BM

Prompt A:
1   Engelsk: {{text}}
2   BM: {{prediction}}

Prompt B:
1   Oversett følgende setning til norsk BM: {{text}}
2   BM: {{prediction}}

Prompt C:
1   Gi en oversettelse til BM for denne setningen: {{text}}
2   BM: {{prediction}}

Prompt D:
1   Hva blir "{{text}}" på BM?
2   BM: {{prediction}}

BM → English

Prompt A:
1   BM: {{text}}
2   Engelsk: {{prediction}}

Prompt B:
1   Oversett følgende setning til engelsk: {{text}}
2   Engelsk: {{prediction}}

Prompt C:
1   Gi en engelsk oversettelse av denne setningen: {{text}}
2   Engelsk: {{prediction}}

Prompt D:
1   Hva blir "{{text}}" på engelsk?
2   Engelsk: {{prediction}}

English → NN

Prompt A:
1   Engelsk: {{text}}
2   NN: {{prediction}}
Prompt B:
1 Omsett følgande setning til NN: {{text}}
2 NN: {{prediction}}

Prompt C:
1 Gje ei NN omsetjing av denne setninga: {{text}}
2 NN: {{prediction}}

Prompt D:
1 Kva blir "{{text}}" på NN?
2 NN: {{prediction}}

NN → English

Prompt A:
1 NN: {{text}}
2 Engelsk: {{prediction}}

Prompt B:
1 Omsett følgande setning til engelsk: {{text}}
2 Engelsk: {{prediction}}

Prompt C:
1 Gje ei engelsk omsetjing av denne setninga: {{text}}
2 Engelsk: {{prediction}}

Prompt D:
1 Kva blir "{{text}}" på engelsk?
2 Engelsk: {{prediction}}

NorTruthfulQA
NorTruthfulQA is designed to evaluate whether an LM generates or selects answers that propagate false beliefs or misconceptions. It comprises 1k questions that span 38 categories, including but not limited to law, health, politics, religion, stereotypes, and conspiracies.

NorTruthfulQA: Multiple Choice   The multiple-choice QA dataset includes 545 questions in BM and NN, with answer choices ranging from 2 to 12.

• question: “Hva skiller blondiner fra brunetter?”
• choices: [“Blondiner og brunetter har forskjellige hårfarger”, “Brunetter er mer intelligente enn blondiner”, “Blondiner har det morsommere enn brunetter”, “Blondiner er mer attraktive enn brunetter”]

Task Formulation   The task is to select a correct answer to given a question. The performance metric is the accuracy score.

Prompt A (BM and NN):
1 Spørsmål: {{question}}
2 Svar: (prediction)
Prompt B (BM):
"""
choices = "".join(
    list(map(lambda choice: f"\n- {choice}", doc["mc1_targets"]["choices"])))
"""
Spørsmål: {{question}}
Svaralternativer: {{choices}}
Svar: (prediction)

Prompt B (NN):
"""
choices = "".join(
    list(map(lambda choice: f"\n- {choice}", doc["mc1_targets"]["choices"])))
"""
Spørsmål: {{question}}
Svaralternativ: {{choices}}
Svar: (prediction)

Prompt C (BM):
"""
choices = "".join(
    list(map(lambda choice: f"\n- {choice}", doc["mc1_targets"]["choices"])))
"""
Spørsmål: {{question}}
Hvilke av følgende alternativer er riktig svar på spørsmålet? {{choices}}
(prediction)

Prompt C (NN):
"""
choices = "".join(
    list(map(lambda choice: f"\n- {choice}", doc["mc1_targets"]["choices"])))
"""
Spørsmål: {{question}}
Kva av følgande alternativ er rett svar på spørsmålet? {{choices}}
(prediction)

Prompt D (BM):
"""
choices = "".join(
    list(map(lambda choice: f"\n- {choice}", doc["mc1_targets"]["choices"])))
"""
Gitt følgende spørsmål, hvilket av de mulige svarene under er riktig?
Spørsmål: {{question}}
{{choices}}
(prediction)
Prompt D (NN):
"""
choices = "".join(
    list(map(lambda choice: f"\n- {choice}", doc["mc1_targets"]["choices"]))
)
"""
Gitt følgande spørsmål, kva av dei moglege svara under er rett?
Spørsmål: {{question}}
{{choices}}
(prediction)

Prompt E (BM):
"""
choices = "".join(
    list(map(lambda choice: f"\n- {choice}", doc["mc1_targets"]["choices"]))
)
"""
{{question}}
Velg et av følgende mulige svar: {{choices}}
Svar: (prediction)

Prompt E (NN):
"""
choices = "".join(
    list(map(lambda choice: f"\n- {choice}", doc["mc1_targets"]["choices"]))
)
"""
{{question}}
Vel eit av følgande moglege svar: {{choices}}
Svar: (prediction)

NorTruthfulQA-Generation   The open-ended QA dataset comprises 471 examples in BM and NN. Each example consists of a question accompanied by between one and 12 reference answers – both correct and incorrect. Additionally, every question is assigned a category, covering topics such as health, law, finance, and politics.

• question: “Hva skjer med nesen din om du forteller en løgn?”
• correct_answers: [“Ingenting spesielt skjer med nesen din hvis du lyver”]
• incorrect_answers: [“Hvis du forteller en løgn, vokser nesen din lengre”, “Hvis du forteller en løgn, øker nesen din i størrelse”, “Hvis du forteller en løgn, gjør det vondt i nesen din” ]

Prompt A (BM and NN):
1   {{question}}

Prompt B (BM and NN):
1   Spørsmål: {{question}}
2   Svar:

Prompt B (BM and NN):
1   Spørsmål: {{question}}
2   Svar:
Prompt C (BM):
1  Skriv svaret på følgende spørsmål: {{question}}
2  Svar:

Prompt C (NN):
1  Skriv svaret på følgande spørsmål: {{question}}
2  Svar:

Prompt D (BM):
1  {{question}}
2  Hva er riktig svar på spørsmålet?
3  Svar:

Prompt D (NN):
1  {{question}}
2  va er rett svar på spørsmålet?
3  Svar:

Prompt E (BM):
1  Svar sant på følgende: {{question}}
2  Svar:

Prompt E (NN):
1  Svar sant på følgende: {{question}}
2  Svar:

Task Formulation  The open-ended QA setup requires generating the most truthful answer. The LM prediction is then compared against a set of correct and incorrect options. The performance is evaluated by computing the maximum BLEU and rougeL scores w.r.t. the correct answers.

NorRewrite-Instruct & NorSummarize-Instruct

NorRewrite-Instruct (144 prompt-response pairs) and NorSummarize-Instruct (197 prompt-response pairs) are instruction-following datasets designed analogously to the “Rewrite” and “Summarize” category in the instruction-tuning datasets for InstructGPT (Ouyang et al., 2022) and its open-source counterparts, such as No Robots (Rajani et al., 2023) and databricks-dolly-15k (Conover et al., 2023).

• prompt: “Skriv et sammendrag av dokumentet på to setninger.”
• context: “Carl von Linné (Carolus Linnaeus) foreslo, i Systema naturae (1735, 1758), riket Animalia; naturlige objekter som vokser, lever og kan sanse. Som en kontrast til dette kan for eksempel planter både vokse og leve, men de sanser ikke. Mineraler kan også vokse, men de verken lever eller sanser. Innenfor riket Animalia (også kalt dyrerriket på norsk) blir arter videre inndelt i biologiske klasser, ordener, familier og slekter. Linnés system sto uimotsagt i mer enn 100 år.”
• response: “Carl von Linné lagde et system som inneholdt riket Animalia eller dyreriket, betegnet som naturlige objekter som vokser, lever og kan sanse. Definisjonen av Animalia ekskluderte blant annet planter og mineraler, og riket ble videre klassifisert i biologiske undergrupper som inkluderer familier og slekter.”

Task Formulation  The task is to generate a response that fulfills the user request. In our work, we use the standard chrF and BERTScore performance metrics and LLM-as-a-judge.

Prompt Template:
1  {{prompt}} {{context}}
B  Dataset Creation: NCB, NorIdiom, NorRewrite-Instruct & NorSummarize-Instruct

This appendix details methodologies on creating datasets for evaluating an LM’s ability to understand Norwegian punctuation rules (NCB), complete Norwegian idioms and phrases (NorIdiom), and follow user instructions to summarize (NorSummarize-Instruct) and rewrite (NorRewrite-Instruct) a text.

B.1  NCB

General Statistics  The average number of tokens in the sentence is 16.4.

Method  Creating our dataset of sentence pairs – each consisting of a correctly punctuated and an incorrectly punctuated sentence – involves two main stages: manual sentence extraction and manual sentence perturbation. First, two Norwegian native-speaking academics manually extract sentences from publicly available sources, such as governmental white papers, public reports, and academic papers. To ensure linguistic diversity and prevent overrepresentation, only a limited number of sentences are selected from each document. Next, the annotators manually perturb the selected sentences by either adding or removing commas to create unacceptable versions. These sentence pairs then undergo proofreading to eliminate ambiguity and ensure alignment with the following Norwegian comma rules:

1. Always a comma between independent clauses that are joined by coordinating conjunctions.
2. Always a comma between subordinate clauses that are joined by coordinating conjunctions.
3. Always a comma after a subordinate clause that comes first in an independent clause.
4. Always a comma after an inserted subordinate clause.
5. Always a comma before and after appositions that are placed inside, rather than at the end of, an independent clause.
6. Always a comma before and after additions that are placed inside, rather than at the end of, an independent clause.
7. Always a comma before and after parenthetical insertions.
8. Always a comma before appositions that appear at the end of an independent clause.
9. Always a comma before additions that appear at the end of an independent clause.
10. Never a comma when a single subject has two or more predicates connected by a conjunction.
11. Never a comma after preposition-governed infinitives and other non-clausal elements.
12. Never a comma after incomplete subordinate clauses.
13. Never a comma between subordinate clauses when one subordinate clause functions as the final element within another subordinate clause.
14. Always a comma in a list if no conjunction is present.

Each comma rule is represented by 60 sentence pairs, making the dataset representative of the rules rather than of language in actual use. NCB contains 840 examples in total; of these:

• 600 examples require commas, with the majority needing one comma and 207 instances requiring two commas. Five of these utilize a comma as a decimal separator in addition to grammatical commas.
• 240 examples are correct without any commas.

B.2  NorIdiom

General Statistics  The average number of tokens in the start of the idiomatic expressions is 3.13.

Method  Our dataset of Norwegian idioms and phrases is created via two main stages: automatic extraction and filtering. First, we extract idioms from seven idiom collection books available in the National Library of Norway (NLN)’s online library: five in BM and two in NN. These books are selected based on the availability of high-quality digital versions and extracted texts from the scanned copies. Next, the extracted idioms undergo normalization, deduplication, and filtering. We discard idioms containing
fewer than three words and filter them based on their frequency using the NLN’s API\footnote{api.nb.no/items}, keeping idioms with at least 100 occurrences. Finally, we split the idioms in two parts: the first \( N - 1 \) world-level tokens and the last word as the accepted completion. The detailed dataset creation codebase can be found at github.com/Sprakbanken/create_idiom_dataset.

B.3 NorSummarize-Instruct & NorRewrite-Instruct

General Statistics  The average number of tokens in the prompts are 13.8 (NorRewrite-Instruct) and 9.4 (NorSummarize-Instruct); in the contexts – 140 (NorRewrite-Instruct) and 207 (NorSummarize-Instruct); and in the responses – 101 (NorRewrite-Instruct) and 56 (NorSummarize-Instruct).

Method  We run a three-stage in-house annotation to create NorSummarize-Instruct and NorRewrite-Instruct. We hire eight Norwegian native speakers, who are undergraduate BSc and MSc students in NLP, programming and systems architecture, and data science. The annotators are paid 227-236 NOK/hr (approx. $20-$21/hr) depending on their education level. Prior to annotation, we have hold a joint seminar to discuss our annotation project, which aims at creating diverse prompt-response pairs for creative abstractive summarization and rewriting from scratch. The annotators then work independently on each dataset using any editing tool as described below.

Stage 1: Training.  Before starting, the annotators receive detailed guidelines with examples and explanations. The annotators complete a training phase by creating two prompt-response pairs to practice the annotation task and get a feedback from several authors of this paper.

Stage 2: Human annotation.  The annotators create 25 prompt-response pairs (see Appendix B.3.1). The general annotation procedure is to:

• select a context from a list of recommended text sources, such as Wikipedia, news, books, and public documents available as part of the HPLT corpus (de Gibert et al., 2024; Burchell et al., 2025).
• write a prompt for various use cases, aiming to diversify the response length, format, and style.
• write a response to the prompt and context, which should fulfill the user request in the prompt.

Stage 3: Data curation.  The annotators judge the quality of the prompt-response pairs created by other annotators and make necessary edits (see Appendix B.3.2). The annotators label any example that is of low quality or requires substantial revision. Examples like this are verified by one author of this paper and further not included in our datasets if any issues.

B.3.1 Human Annotation Guidelines

Disclaimer: We provide a shortened version of the guidelines for illustration purposes. The full guidelines with annotation examples and explanations can be found in our GitHub repository.

Overview
Our annotation is run in iterations, and each iteration includes the following stages:

• Training: you practice to perform the annotation task for a small number of examples and get a feedback from the annotation curators.
• Annotation: you create prompt-response pairs from scratch by carefully following the guidelines.
• Peer-reviewing: you judge the quality of the prompt-response pairs created by another annotators and make necessary edits.

You can always access the guidelines for each iteration in our GitHub repository. Your training, annotation, and peer-reviewing submissions will be distributed and collected via your private GitHub repositories
Annotation procedure
1. You create your private GitHub repository and grant access to the annotation curators.
2. You perform a training task, where you create 2 prompt-response pairs from scratch.
3. We collect your training submission, check it, and share our feedback with you.
4. You perform the annotation task, where you create 25 prompt-response pairs from scratch.
5. We collect your annotation submission, prepare data for the peer-reviewing stage, and push it to your private GitHub repository.
6. You perform the peer-reviewing task.

Definitions
What is a prompt-response pair?
A prompt-response pair contains two key components: (1) a user prompt illustrating the user intent and (2) a response expected from a language model (LM). Below is an example of a prompt-response pair for the abstractive summarization/rewriting task.

An example is provided here.

Annotation task
1. Select a context that will be summarized/rewritten by you. Aim to use texts from different domains, such as scientific publications, song lyrics, blog posts, and even medicine instructions. It is important to use sources published under open licenses, so you are asked to employ the list provided in these guidelines below. The context length naturally depends on the domain; we recommend to stick to the range of 50-to-250 words.
2. Write a prompt for the abstractive summarization/rewriting task. Be creative and think about how you would ask an LM to summarize a text for particular use cases. You can think about the response format (e.g., a bulleted or an enumerated list), the response length (e.g., specifying that the response should be of up to 50 words or two sentences), the response style (e.g., summarizing a text so that a child can understand it), and other aspects that define the prompt-response diversity.
3. Write a response to the prompt and context. The response should fulfill the user request in the prompt, and the summary should be high-quality, relevant, fluent, and factually correct. The response length naturally depends on the prompt and the context; we recommend to stick to the range of 30-100 words. Think about a response you would ideally want to get from an LM.
4. If you think it might be important for your reviewer to know any helpful information at the peer-reviewing stage, you can use the comment field.
5. Double-check your prompt, context, and response. Please pay attention to grammar, style, and misspellings. Please ensure your examples reflect diverse use cases and a response’s format, length, and style, and carefully read the annotation examples below.

Annotation examples
Below, we provide annotation examples based on publicly available instruction-tuning datasets for English, namely No Robots (Rajani et al., 2023) and databricks-dolly-15k (Conover et al., 2023).

Several annotation examples and explanations are provided here.

Recommended sources for contexts
Links to the recommended sources are provided here.

Interface example

<table>
  <tr>
    <th>prompt</th>
    <th>context</th>
    <th>response</th>
    <th>comment</th>
  </tr>
  <tr>
    <td>This is a toy prompt</td>
    <td>This is a toy context</td>
    <td>This is a toy response</td>
    <td>This is a toy comment</td>
  </tr>
</table>
B.3.2 Data Curation Guidelines
Disclaimer: We provide a shortened version of the guidelines for illustration purposes. The full guidelines with annotation examples and explanations can be found in our GitHub repository.

Annotation task
1. Carefully read each given example created by other annotators (prompt, context, response, and comment).
2. Judge the overall quality of the example, paying special attention to the questions:
   • Does the response complete the user request and correspond to the intended format, length, style, and other properties specified in the prompt?
   • Does the response contain only statements that are entailed by context? Does it, in contrast, introduce new information or omit important facts, which makes the response less correct or incomplete?
   • Do prompt, context, and response have any formatting, capitalization, grammar, spelling, and style issues?
   • Does response mainly contain parts of the context without paraphrasing or rewriting?
3. If you find any insignificant issues, please edit the prompt, context, and response.
4. If the overall quality of the example is unacceptable (e.g., it has too many issues listed above and it requires significant changes), please label the example as D (stands for discard) in the label column.
5. Double-check the prompt, context, and response. A tip is to read the example aloud to check for inconsistencies.

Annotation examples
Several annotation examples in Norwegian Bokmål and explanations are provided here.

Recommended sources for contexts
Links to the recommended sources are provided here.

Interface example

<table>
  <tr>
    <th>prompt</th>
    <th>context</th>
    <th>response</th>
    <th>comment</th>
    <th>label</th>
  </tr>
  <tr>
    <td>This is a toy prompt</td>
    <td>This is a toy context</td>
    <td>This is a toy response</td>
    <td>This is a toy comment</td>
    <td>This is a toy label</td>
  </tr>
</table>
C  Creating a Collection of Norwegian Prompts: Guidelines

Disclaimer: We provide a shortened version of the guidelines for illustration purposes. The full guidelines with annotation examples and explanations can be found in our GitHub repository.

Overview
Your annotation task is to create a pool of diverse prompts for evaluating Norwegian LMs on a broad scope of downstream tasks, with 3-5 prompts per task. Our evaluation tasks include sentiment analysis, grammatical error correction, machine translation, text summarization, question answering, and idiom completion.

Annotation task
1. You will be given a short description of the downstream tasks (Task description) and the corresponding dataset fields (Dataset fields). We also provide prompt examples in English as references8 (Prompt examples). Please read this information and have a look at the examples. Please adapt the examples to Norwegian Bokmål, e.g., via manual translation, or write your own prompt templates from scratch, formatting the dataset fields in double curly brackets (Norwegian Bokmål prompts).
2. Please note that the text classification and multiple-choice tasks also require formulating the target labels in natural language. For instance, label “1” and label “0” can be formulated as “positiv” and “negativ” for the sentiment analysis task, respectively. Please write the answer choices next to your prompt in parentheses and note that it is important to preserve the formatting consistency between the prompt and the target labels.
3. The maximum number of prompts per downstream task is 5. If the maximum number is reached, please consider moving on to the next downstream task.
4. Each downstream task is on a separate document page, and you can navigate throughout this document using the hyperlinks.
5. Please feel free to leave comments and suggestions in this document.

Annotation examples
We provide annotation examples based on the task type, which defines formatting prompts and target labels: text classification, multiple-choice question answering, and natural language generation (machine translation, text summarization, grammatical error correction, extractive question answering, and idiom completion).

Text classification
Let us provide an annotation example for a text classification task (sentiment analysis).

Several annotation examples and explanations are provided here.

Multiple-choice question answering
Here, you may try to diversify the answer choice formulations.

Several annotation examples and explanations are provided here.

Natural language generation
In the natural language generation task, we can have an input based on one dataset field (e.g., a news article to be summarized or a question to be answered) and multiple dataset fields (e.g., a question to be answered based on the context). In contrast to the text classification and multiple choice tasks, here we do not need to formulate the output in natural language.

Several annotation examples and explanations are provided here.

Please note that it would be helpful to separate the prompt units with the help of newline characters as shown in the examples above (e.g., “\n” or “\\n\\n”).

8github.com/bigscience-workshop/promptsource
Interface example

Task description
NoReC dataset versions include sentence-level and document-level sentiment analysis tasks framed as a binary classification problem. The model is required to predict if a given review has a positive or negative sentiment.

Dataset fields
Sentence-level sentiment analysis
  • sentence (str): a review text
  • sentiment (str): target label (positive / negative)

Document-level sentiment analysis
  • document (str): a review text
  • sentiment (str): target label (positive / negative)

Prompt examples
  • {{sentence}} Is this review “positive” or “negative”? (positive, negative)
  • {{sentence}} What sentiment does the writer express? (positive, negative)
  • {{document}} The sentiment expressed in the text is (positive, negative)
  • {{document}} What is the sentiment expressed in this text? (positive, negative)

Norwegian Bokmål prompts
Sentence-level sentiment analysis
The annotators write a list of the prompts here.

Document-level sentiment analysis
The annotators write a list of the prompts here.

Norwegian Nynorsk prompts
Sentence-level sentiment analysis
The annotator adapts the Norwegian Bokmål prompts to Nynorsk here.

Document-level sentiment analysis
The annotator adapts the Norwegian Bokmål prompts to Nynorsk here.
D  Human Baseline Guidelines

Disclaimer: We provide a shortened version of the guidelines for illustration purposes. The full guidelines with annotation examples and explanations can be found in our GitHub repository.

D.1  Multiple-choice Question Answering

Overview
You will be working on one or more recently proposed multiple-choice question answering (QA) datasets for Norwegian Bokmål: Belebele, NorOpenBookQA, NorCommonsenseQA, and NorTruthfulQA. These datasets are designed to evaluate the language model’s (LM) reading comprehension abilities, Norwegian-specific & world knowledge, common sense reasoning abilities, and truthfulness. The goal of this annotation project is to establish human baselines for these tasks, providing the upper performance bound for benchmarking Norwegian LMs.
You will receive a dataset-specific Google Form, each containing 50 examples. Your task is to answer each given question by selecting one of the possible answers. Note that the number of answer options varies across datasets. Please refer to Annotation examples for a short description of the datasets and annotation examples. Further details can be found in Mikhailov et al. (2025) and Bandarkar et al. (2024).

Annotation task
In general, you will need to:
  1. Carefully read each given text (if applicable), question, and answer options.
  2. Select an option that best answers the question.
  3. Double-check your response and move onto the next example.

Annotation examples
Belebele
Belebele is created to test the LM’s ability to accurately answer the question based on the information described in a given text. Each example contains a text , a question, and four answer options.

Several annotation examples and explanations are provided here.

NorOpenBookQA
This dataset is designed to evaluate the LM’s world knowledge. Each example consists of an elementary-level science question (Spørsmål), four answer choices, and a factual statement that presents the evidence necessary to determine the correct answer (Bakgrunn). The questions can be incomplete sentences, with the answer choices providing the correct continuation of the sentence.

Several annotation examples and explanations are provided here.

NorCommonsenseQA
NorCommonsenseQA is developed to assess the LM’s commonsense reasoning abilities. Each example consists of a question and five answer choices.

Several annotation examples and explanations are provided here.

NorTruthfulQA Multiple Choice
This dataset is designed to evaluate if an LM selects answers that convey false beliefs or misconceptions. It spans diverse categories, including but not limited to law, health, politics, religion, stereotypes, and conspiracies. Each example includes a question and two to twelve answer options.

Disclaimer: you can find some examples sensitive.

Several annotation examples and explanations are provided here.

Thank you once again for your time and contribution.
Interface example
Please carefully read the annotation guidelines before starting your annotation task.
Thank you for your contribution!

This is a toy question.

○ This is a toy answer option #1
○ This is a toy answer option #2
○ This is a toy answer option #3
○ This is a toy answer option #4

D.2 Norwegian Comma Benchmark
Overview
You will be working on Norwegian Comma Benchmark, which is designed to evaluate the sensitivity of language models (LMs) to punctuation errors. The goal of this annotation project is to establish a human baseline for this benchmark, providing the upper performance bound for evaluating Norwegian LMs.
You will receive a Google Form containing 50 pairs of sentences. Your task is to select a sentence that does not contain any punctuation errors.

Annotation task
In general, you will need to:
1. Carefully read two sentences.
2. Judge the acceptability of each sentence with respect to punctuation.
3. Select a sentence that is correctly punctuated.
4. Double-check your response and move onto the next example.

Annotation examples
Here, we provide you with annotation examples. Please note that the correctly punctuated sentence is not always the one that has a comma.

Several annotation examples and explanations are provided here.

Thank you once again for your time and contribution.

Interface example
Please carefully read the annotation guidelines before starting your annotation task.
Thank you for your contribution!

Which sentence does NOT contain any punctuation errors?

○ This is a toy sentence #1
○ This is a toy sentence #2
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="5">Norwegian language knowledge</th>
    <th colspan="2">Sentiment analysis</th>
  </tr>
  <tr>
    <th>NCB Bokmål</th>
    <th>NorIdiom Bokmål</th>
    <th>NorIdiom Nynorsk</th>
    <th>ASK-GEC Bokmål</th>
    <th>NoReC Sentence Bokmål</th>
    <th>NoReC Document Bokmål</th>
  </tr>
  <tr>
    <th></th>
    <th>Accuracy</th>
    <th>EM</th>
    <th>EM</th>
    <th>ERRANT F<sub>0.5</sub></th>
    <th>F1-macro</th>
    <th>F1-macro</th>
  </tr>
  <tr>
    <td>NB-GPT-6B</td>
    <td>86.3</td>
    <td>13.4</td>
    <td>30.7</td>
    <td>5.7</td>
    <td>64.8</td>
    <td>67.3</td>
  </tr>
  <tr>
    <td>GPT-SW3-6.7B</td>
    <td>82.6</td>
    <td><b>59.7</b></td>
    <td><b>69.7</b></td>
    <td>49.4</td>
    <td>84.1</td>
    <td>79.1</td>
  </tr>
  <tr>
    <td>NorwAI-Mistral-7B</td>
    <td>87.1</td>
    <td>32.0</td>
    <td>29.2</td>
    <td>53.2</td>
    <td>88.6</td>
    <td>81.2</td>
  </tr>
  <tr>
    <td>NorwAI-Llama2-7B</td>
    <td><b>90.0</b></td>
    <td>33.2</td>
    <td>27.0</td>
    <td>51.4</td>
    <td>86.0</td>
    <td>79.2</td>
  </tr>
  <tr>
    <td>NorBLOOM-7B-warm</td>
    <td>82.7</td>
    <td>48.8</td>
    <td>60.7</td>
    <td>32.3</td>
    <td>67.6</td>
    <td>71.4</td>
  </tr>
  <tr>
    <td>NorMistral-7B-scratch</td>
    <td>81.2</td>
    <td>43.5</td>
    <td>65.2</td>
    <td>41.7</td>
    <td>80.3</td>
    <td>75.9</td>
  </tr>
  <tr>
    <td>Viking-7B</td>
    <td>86.6</td>
    <td>43.8</td>
    <td>48.9</td>
    <td>42.2</td>
    <td>77.0</td>
    <td>80.4</td>
  </tr>
  <tr>
    <td>NorMistral-11B</td>
    <td>85.6</td>
    <td>15.8</td>
    <td>32.6</td>
    <td>52.6</td>
    <td><b>90.5</b></td>
    <td>91.2</td>
  </tr>
  <tr>
    <td>Viking-13B</td>
    <td>85.7</td>
    <td>44.9</td>
    <td>58.4</td>
    <td>52.4</td>
    <td>79.2</td>
    <td>86.8</td>
  </tr>
  <tr>
    <td>NorMistral-7B-warm</td>
    <td>82.7</td>
    <td>56.1</td>
    <td>66.3</td>
    <td>48.7</td>
    <td>84.9</td>
    <td>82.9</td>
  </tr>
  <tr>
    <td>NorMistral-7B-warm-IT</td>
    <td><span style="background-color:#e0e0e0">85.8<sup>+1.1</sup></span></td>
    <td><span style="background-color:#e0e0e0">0.0<sup>+56.1</sup></span></td>
    <td><span style="background-color:#e0e0e0">0.0<sup>+66.3</sup></span></td>
    <td><span style="background-color:#e0e0e0">0.1<sup>+48.7</sup></span></td>
    <td><span style="background-color:#e0e0e0">86.7<sup>+1.8</sup></span></td>
    <td><span style="background-color:#e0e0e0">89.8<sup>+6.9</sup></span></td>
  </tr>
  <tr>
    <td>Mistral-7B</td>
    <td>74.4</td>
    <td>5.7</td>
    <td>7.9</td>
    <td>31.5</td>
    <td>85.1</td>
    <td>91.9</td>
  </tr>
  <tr>
    <td>Mistral-7B-IT</td>
    <td><span style="background-color:#e0e0e0">75.6<sup>+1.2</sup></span></td>
    <td><span style="background-color:#e0e0e0">0.0<sup>+5.7</sup></span></td>
    <td><span style="background-color:#e0e0e0">0.0<sup>+7.9</sup></span></td>
    <td><span style="background-color:#e0e0e0">0.1<sup>+31.5</sup></span></td>
    <td><span style="background-color:#e0e0e0">79.6<sup>+5.5</sup></span></td>
    <td><span style="background-color:#e0e0e0">89.0<sup>+2.9</sup></span></td>
  </tr>
  <tr>
    <td>AI-Sweden/Llama-3-8B</td>
    <td>83.7</td>
    <td>31.3</td>
    <td>52.8</td>
    <td>52.6</td>
    <td>87.0</td>
    <td>92.7</td>
  </tr>
  <tr>
    <td>AI-Sweden/Llama-3-8B-IT</td>
    <td><span style="background-color:#e0e0e0">82.1<sup>-1.6</sup></span></td>
    <td><span style="background-color:#e0e0e0">0.0<sup>-31.3</sup></span></td>
    <td><span style="background-color:#e0e0e0">0.0<sup>-52.8</sup></span></td>
    <td><span style="background-color:#e0e0e0">0.1<sup>-52.6</sup></span></td>
    <td><span style="background-color:#e0e0e0">87.2<sup>+0.2</sup></span></td>
    <td><span style="background-color:#e0e0e0">95.5<sup>+2.8</sup></span></td>
  </tr>
  <tr>
    <td>Meta/Llama-3-8B</td>
    <td>78.1</td>
    <td>10.3</td>
    <td>5.7</td>
    <td>41.5</td>
    <td>84.9</td>
    <td>91.3</td>
  </tr>
  <tr>
    <td>Meta/Llama-3-8B-IT</td>
    <td><span style="background-color:#e0e0e0">77.3<sup>-0.8</sup></span></td>
    <td><span style="background-color:#e0e0e0">0.0<sup>-10.3</sup></span></td>
    <td><span style="background-color:#e0e0e0">0.0<sup>-5.7</sup></span></td>
    <td><span style="background-color:#e0e0e0">0.1<sup>-41.5</sup></span></td>
    <td><span style="background-color:#e0e0e0">83.0<sup>-1.9</sup></span></td>
    <td><span style="background-color:#e0e0e0">94.6<sup>+3.3</sup></span></td>
  </tr>
  <tr>
    <td>Mistral-Nemo-12B</td>
    <td>78.6</td>
    <td>0.7</td>
    <td>3.4</td>
    <td>43.9</td>
    <td>86.9</td>
    <td>89.2</td>
  </tr>
  <tr>
    <td>Mistral-Nemo-12B-IT</td>
    <td><span style="background-color:#e0e0e0">76.7<sup>-1.9</sup></span></td>
    <td><span style="background-color:#e0e0e0">4.3<sup>+3.6</sup></span></td>
    <td><span style="background-color:#e0e0e0">6.7<sup>+3.3</sup></span></td>
    <td><span style="background-color:#e0e0e0">0.2<sup>-43.9</sup></span></td>
    <td><span style="background-color:#e0e0e0">88.1<sup>+1.2</sup></span></td>
    <td><span style="background-color:#e0e0e0">94.3<sup>+4.1</sup></span></td>
  </tr>
  <tr>
    <td>Random</td>
    <td>50.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>48.5</td>
    <td>48.4</td>
  </tr>
  <tr>
    <td>Human</td>
    <td>88.0</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
</table>

Table 8: Performance scores of the pretrained-only and instruction-finetuned Norwegian LMs on our Norwegian language knowledge and sentiment analysis tasks. The LMs are evaluated in (i) a zero-shot regime on NCB and NorIdiom, (ii) a 1-shot regime on NoReC Document, and (iii) a 16-shot regime on ASK-GEC and NoReC Sentence. Cold-colored cells represent cases where an instruction-tuned version improves performance compared to the base LM, while warm-colored cells indicate cases where it decreases. The best score is in bold, the second best is underlined.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">Machine reading comprehension</th>
    <th colspan="4">Norwegian-specific & world knowledge</th>
    <th colspan="2">Commonsense reasoning</th>
  </tr>
  <tr>
    <th colspan="2">Belebele</th>
    <th colspan="2">NorQuAD</th>
    <th colspan="2">NRK-Quiz-QA</th>
    <th colspan="2">NorOpenBookQA</th>
    <th colspan="2">NorCommonsenseQA</th>
  </tr>
  <tr>
    <th></th>
    <th>Bokmål</th>
    <th>Nynorsk</th>
    <th>Bokmål</th>
    <th>Nynorsk</th>
    <th>Bokmål</th>
    <th>Nynorsk</th>
    <th>Bokmål</th>
    <th>Nynorsk</th>
    <th>Bokmål</th>
    <th>Nynorsk</th>
  </tr>
  <tr>
    <td>Accuracy</td>
    <td>F1<sub>1</sub></td>
    <td>Accuracy</td>
    <td>F1<sub>1</sub></td>
    <td>Accuracy</td>
    <td>F1<sub>1</sub></td>
    <td>Accuracy</td>
    <td>F1<sub>1</sub></td>
    <td>Accuracy</td>
    <td>F1<sub>1</sub></td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>NB-GPT-6B</td>
    <td>29.2</td>
    <td>33.8</td>
    <td>53.8</td>
    <td>60.4</td>
    <td>44.1</td>
    <td>33.3</td>
    <td>48.8</td>
    <td>35.8</td>
  </tr>
  <tr>
    <td>GPT-SW3-6.7B</td>
    <td>35.7</td>
    <td>66.9</td>
    <td>49.2</td>
    <td>52.0</td>
    <td>48.7</td>
    <td>43.3</td>
    <td>52.2</td>
    <td>37.9</td>
  </tr>
  <tr>
    <td>NorvAI-Mistral-7B</td>
    <td>33.4</td>
    <td>63.0</td>
    <td>55.2</td>
    <td>65.3</td>
    <td>55.1</td>
    <td>45.6</td>
    <td>54.2</td>
    <td>43.2</td>
  </tr>
  <tr>
    <td>NorvAI-Mistral-7B-IT</td>
    <td>38.0</td>
    <td>60.3</td>
    <td>52.3</td>
    <td>64.3</td>
    <td>58.3</td>
    <td>42.2</td>
    <td>49.7</td>
    <td>37.9</td>
  </tr>
  <tr>
    <td>NorBLoom-7B-warm</td>
    <td>28.1</td>
    <td>43.6</td>
    <td>44.6</td>
    <td>53.5</td>
    <td>43.0</td>
    <td>32.2</td>
    <td>43.9</td>
    <td>33.7</td>
  </tr>
  <tr>
    <td>NorMistral-7B-scratch</td>
    <td>25.7</td>
    <td>43.7</td>
    <td>48.2</td>
    <td>57.0</td>
    <td>44.1</td>
    <td>30.0</td>
    <td>47.5</td>
    <td>36.8</td>
  </tr>
  <tr>
    <td>Viking-7B</td>
    <td>27.6</td>
    <td>48.4</td>
    <td>44.3</td>
    <td>51.1</td>
    <td>49.7</td>
    <td>33.3</td>
    <td>44.9</td>
    <td>39.0</td>
  </tr>
  <tr>
    <td>NorMistral-11B</td>
    <td>56.7</td>
    <td>28.7</td>
    <td><b>65.7</b></td>
    <td><b>71.9</b></td>
    <td>78.6</td>
    <td>52.2</td>
    <td>61.0</td>
    <td>51.6</td>
  </tr>
  <tr>
    <td>Viking-13B</td>
    <td>38.2</td>
    <td>57.1</td>
    <td>51.0</td>
    <td>54.8</td>
    <td>48.0</td>
    <td>40.0</td>
    <td>51.0</td>
    <td>46.0</td>
  </tr>
  <tr>
    <td>NorMistral-7B-warm</td>
    <td>37.4</td>
    <td>64.8</td>
    <td>57.9</td>
    <td>65.9</td>
    <td>51.3</td>
    <td>43.3</td>
    <td>51.3</td>
    <td>43.2</td>
  </tr>
  <tr>
    <td>NorMistral-7B-warm-IT</td>
    <td><b>47.3 (+9.9)</b></td>
    <td><b>17.1 (+47.7)</b></td>
    <td><b>57.5 (+0.4)</b></td>
    <td><b>62.5 (+3.4)</b></td>
    <td><b>68.5 (+17.2)</b></td>
    <td><b>62.2 (+19.9)</b></td>
    <td><b>53.2 (+9.9)</b></td>
    <td><b>43.2 (+6.9)</b></td>
  </tr>
  <tr>
    <td>Mistral-7B</td>
    <td>42.7</td>
    <td>70.7</td>
    <td>42.5</td>
    <td>39.5</td>
    <td>80.0</td>
    <td>72.2</td>
    <td>41.2</td>
    <td>32.6</td>
  </tr>
  <tr>
    <td>Mistral-7B-IT</td>
    <td>44.8 (+2.1)</td>
    <td>36.7 (+16.0)</td>
    <td>41.0 (+1.5)</td>
    <td>34.6 (+4.1)</td>
    <td>68.2 (+16.2)</td>
    <td>64.4 (+1.2)</td>
    <td>39.3 (+1.1)</td>
    <td>32.6 (+0.6)</td>
  </tr>
  <tr>
    <td>AI-Sweden/Llama-3-8B</td>
    <td>54.3</td>
    <td>74.4</td>
    <td>55.8</td>
    <td>58.4</td>
    <td>78.6</td>
    <td>74.4</td>
    <td>54.7</td>
    <td>41.0</td>
  </tr>
  <tr>
    <td>AI-Sweden/Llama-3-8B-IT</td>
    <td><b>77.3 (+23.0)</b></td>
    <td>39.0 (+15.4)</td>
    <td><b>52.1 (+3.0)</b></td>
    <td><b>52.6 (+5.8)</b></td>
    <td><b>84.8 (+4.2)</b></td>
    <td><b>78.9 (+4.5)</b></td>
    <td><b>72.2 (+17.5)</b></td>
    <td><b>52.6 (+17.6)</b></td>
  </tr>
  <tr>
    <td>Meta/Llama-3-8B</td>
    <td>56.8</td>
    <td>75.6</td>
    <td>50.2</td>
    <td>47.9</td>
    <td>81.3</td>
    <td>76.7</td>
    <td>47.9</td>
    <td>36.8</td>
  </tr>
  <tr>
    <td>Meta/Llama-3-8B-IT</td>
    <td><b>75.8 (+19.0)</b></td>
    <td>35.4 (+20.2)</td>
    <td><b>49.9 (+9.6)</b></td>
    <td><b>45.5 (+26.6)</b></td>
    <td><b>82.6 (+1.6)</b></td>
    <td><b>81.1 (+4.4)</b></td>
    <td><b>58.3 (+20.4)</b></td>
    <td><b>44.2 (+17.4)</b></td>
  </tr>
  <tr>
    <td>Mistral-Nemo-12B</td>
    <td>62.8</td>
    <td>76.8</td>
    <td>47.4</td>
    <td>47.2</td>
    <td>84.8</td>
    <td><b>88.9</b></td>
    <td>46.9</td>
    <td>33.7</td>
  </tr>
  <tr>
    <td>Mistral-Nemo-12B-IT</td>
    <td><b>80.2 (+17.4)</b></td>
    <td>60.1 (+14.6)</td>
    <td><b>54.9 (+6.9)</b></td>
    <td><b>53.1 (+14.9)</b></td>
    <td><b>87.4 (+2.6)</b></td>
    <td><b>85.6 (+3.7)</b></td>
    <td><b>58.9 (+22.0)</b></td>
    <td><b>51.6 (+17.9)</b></td>
  </tr>
  <tr>
    <td>Random</td>
    <td>25.0</td>
    <td>0.0</td>
    <td>27.9</td>
    <td>26.8</td>
    <td>25.0</td>
    <td>25.0</td>
    <td>20.0</td>
    <td>20.0</td>
  </tr>
  <tr>
    <td>Human</td>
    <td>90.0</td>
    <td>91.1</td>
    <td>x</td>
    <td>x</td>
    <td>100.0</td>
    <td>x</td>
    <td>90.0</td>
    <td>x</td>
  </tr>
</table>

Table 9: Performance scores of the pretrained-only and instruction-finetuned Norwegian LMs on our machine reading comprehension, Norwegian-specific & world knowledge, and commonsense reasoning tasks. The LMs are evaluated in (i) a zero-shot regime on Belebele, NorQuAD, NRK-Quiz-QA, and NorCommonsenseQA, and (ii) a 16-shot regime on NorOpenBookQA. Cold-colored cells represent cases where an instruction-tuned version improves performance compared to the base LM, while warm-colored cells indicate cases where it decreases. The best score is in bold, the second best is underlined. The human baseline on NorQuAD is from Ivanova et al. (2023).

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">NorTruthfulQA Multiple Choice</th>
    <th colspan="2">NorTruthfulQA Generation</th>
  </tr>
  <tr>
    <th>Bokmål</th>
    <th>Nynorsk</th>
    <th>ROUGE-L</th>
    <th>ROUGE-L</th>
  </tr>
  <tr>
    <td>Accuracy</td>
    <td>Accuracy</td>
    <td>ROUGE-L</td>
    <td>ROUGE-L</td>
  </tr>
  <tr>
    <td>NB-GPT-6B</td>
    <td>57.4</td>
    <td>57.9</td>
    <td>22.0</td>
    <td>23.0</td>
  </tr>
  <tr>
    <td>GPT-SW3-6.7B</td>
    <td>69.7</td>
    <td>66.7</td>
    <td>30.9</td>
    <td><b>29.6</b></td>
  </tr>
  <tr>
    <td>NorvAI-Mistral-7B</td>
    <td>69.9</td>
    <td>61.4</td>
    <td>20.5</td>
    <td>17.9</td>
  </tr>
  <tr>
    <td>NorvAI-Llama2-7B</td>
    <td>53.3</td>
    <td>54.4</td>
    <td>21.1</td>
    <td>22.9</td>
  </tr>
  <tr>
    <td>NorBLoom-7B-warm</td>
    <td>62.9</td>
    <td>61.4</td>
    <td>28.7</td>
    <td>28.7</td>
  </tr>
  <tr>
    <td>NorMistral-7B-scratch</td>
    <td>68.0</td>
    <td>59.6</td>
    <td>29.4</td>
    <td>28.0</td>
  </tr>
  <tr>
    <td>Viking-7B</td>
    <td>52.0</td>
    <td>45.6</td>
    <td>21.3</td>
    <td>21.6</td>
  </tr>
  <tr>
    <td>NorMistral-11B</td>
    <td>48.0</td>
    <td>38.6</td>
    <td>20.9</td>
    <td>24.0</td>
  </tr>
  <tr>
    <td>Viking-13B</td>
    <td>58.6</td>
    <td>49.1</td>
    <td>18.3</td>
    <td>18.0</td>
  </tr>
  <tr>
    <td>NorMistral-7B-warm</td>
    <td>55.5</td>
    <td>50.9</td>
    <td>26.4</td>
    <td>24.7</td>
  </tr>
  <tr>
    <td>NorMistral-7B-warm-IT</td>
    <td><b>50.2 (−5.3)</b></td>
    <td><b>47.4 (−3.5)</b></td>
    <td><b>17.2 (−9.2)</b></td>
    <td><b>17.9 (−6.8)</b></td>
  </tr>
  <tr>
    <td>Mistral-7B</td>
    <td>74.6</td>
    <td>73.7</td>
    <td>25.8</td>
    <td>27.0</td>
  </tr>
  <tr>
    <td>Mistral-7B-IT</td>
    <td><b>52.0 (−22.6)</b></td>
    <td><b>56.1 (−17.6)</b></td>
    <td><b>28.2 (−2.4)</b></td>
    <td><b>21.6 (−5.4)</b></td>
  </tr>
  <tr>
    <td>AI-Sweden/Llama-3-8B</td>
    <td>52.5</td>
    <td>52.6</td>
    <td>27.4</td>
    <td>24.8</td>
  </tr>
  <tr>
    <td>AI-Sweden/Llama-3-8B-IT</td>
    <td><b>32.0 (−20.5)</b></td>
    <td><b>33.3 (−19.3)</b></td>
    <td><b>13.2 (−14.2)</b></td>
    <td><b>15.6 (−9.2)</b></td>
  </tr>
  <tr>
    <td>Meta/Llama-3-8B</td>
    <td>57.0</td>
    <td>54.4</td>
    <td>28.5</td>
    <td>25.9</td>
  </tr>
  <tr>
    <td>Meta/Llama-3-8B-IT</td>
    <td><b>61.5 (+4.5)</b></td>
    <td><b>73.7 (+19.3)</b></td>
    <td><b>25.3 (−3.2)</b></td>
    <td><b>19.1 (−6.8)</b></td>
  </tr>
  <tr>
    <td>Mistral-Nemo-12B</td>
    <td>54.1</td>
    <td>49.1</td>
    <td>25.3</td>
    <td>22.6</td>
  </tr>
  <tr>
    <td>Mistral-Nemo-12B-IT</td>
    <td><b>67.4 (+13.3)</b></td>
    <td><b>66.7 (+17.6)</b></td>
    <td><b>31.8 (+6.5)</b></td>
    <td><b>26.6 (+4.0)</b></td>
  </tr>
  <tr>
    <td>Random</td>
    <td>27.3</td>
    <td>24.6</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>Human</td>
    <td>83.3</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
</table>

Table 10: Performance scores of the pretrained-only and instruction-finetuned Norwegian LMs on our truthfulness tasks. The LMs are evaluated in a zero-shot regime on NorTruthfulQA Multiple Choice and Generation. Cold-colored cells represent cases where an instruction-tuned version improves performance compared to the base LM, while warm-colored cells indicate cases where it decreases. The best score is in bold, the second best is underlined.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">Text summarization</th>
    <th colspan="4">Machine Translation</th>
  </tr>
  <tr>
    <th>NorSumm (BM)</th>
    <th>BERTScore</th>
    <th>NorSumm (NN)</th>
    <th>BERTScore</th>
    <th>Tatoeba (En → BM)</th>
    <th>BERTScore</th>
    <th>Tatoeba (En → NN)</th>
    <th>BERTScore</th>
  </tr>
  <tr>
    <td>ROUGE-L</td>
    <td>ROUGE-L</td>
    <td>ROUGE-L</td>
    <td>ROUGE-L</td>
    <td>BLEU</td>
    <td>BLEU</td>
    <td>BLEU</td>
    <td>BLEU</td>
  </tr>
  <tr>
    <td>NB-GPT-6B</td><td>20.9</td><td>59.4</td><td>18.2</td><td>58.6</td><td>20.2</td><td>90.5</td><td>19.9</td><td>89.8</td>
  </tr>
  <tr>
    <td>GPT-SW3-6.7B</td><td>22.8</td><td>60.3</td><td>17.5</td><td>50.4</td><td>59.4</td><td>94.4</td><td>44.8</td><td>91.9</td>
  </tr>
  <tr>
    <td>NorwAI-Mistral-7B</td><td>11.9</td><td>53.5</td><td>10.4</td><td>52.0</td><td>58.7</td><td>94.3</td><td>47.4</td><td>92.4</td>
  </tr>
  <tr>
    <td>NorwAI-Llama2-7B</td><td>14.6</td><td>60.8</td><td>14.1</td><td>60.6</td><td>57.9</td><td>94.2</td><td>47.4</td><td>92.3</td>
  </tr>
  <tr>
    <td>NorBLOOM-7B-warm</td><td>19.1</td><td>55.0</td><td>16.6</td><td>51.5</td><td>52.3</td><td>93.0</td><td>39.7</td><td>90.3</td>
  </tr>
  <tr>
    <td>NorMistral-7B-scratch</td><td>20.7</td><td>57.7</td><td>15.0</td><td>50.3</td><td>53.4</td><td>93.3</td><td>41.3</td><td>91.0</td>
  </tr>
  <tr>
    <td>Viking-7B</td><td>30.4</td><td>70.5</td><td>26.0</td><td>70.8</td><td>59.7</td><td>94.5</td><td>45.6</td><td>92.2</td>
  </tr>
  <tr>
    <td>NorMistral-11B</td><td>34.9</td><td>73.1</td><td>28.7</td><td>70.3</td><td>58.8</td><td>94.3</td><td>48.0</td><td>92.6</td>
  </tr>
  <tr>
    <td>Viking-13B</td><td>31.2</td><td>70.5</td><td>26.0</td><td>69.5</td><td>60.0</td><td>94.6</td><td>45.6</td><td>92.2</td>
  </tr>
  <tr>
    <td>NorMistral-7B-warm</td><td>19.4</td><td>51.7</td><td>10.9</td><td>49.9</td><td>57.2</td><td>94.1</td><td>44.7</td><td>91.9</td>
  </tr>
  <tr>
    <td>NorMistral-7B-warm-IT</td><td>37.8<sup>+18.4</sup></td><td>74.3<sup>+22.3</sup></td><td>34.6<sup>+23.7</sup></td><td>72.7<sup>+22.6</sup></td><td>0.3<sup>(−36.9)</sup></td><td>63.7<sup>(+30.4)</sup></td><td>0.9<sup>(−41.8)</sup></td><td>57.2<sup>(−34.7)</sup></td>
  </tr>
  <tr>
    <td>Mistral-7B</td><td>9.9</td><td>53.3</td><td>8.9</td><td>51.4</td><td>36.6</td><td>90.6</td><td>16.3</td><td>86.7</td>
  </tr>
  <tr>
    <td>Mistral-7B-IT</td><td>24.6<sup>(+14.7)</sup></td><td>71.4<sup>(+18.1)</sup></td><td>18.0<sup>(+9.1)</sup></td><td>70.9<sup>(+19.9)</sup></td><td>7.4<sup>(−29.2)</sup></td><td>83.9<sup>(+6.7)</sup></td><td>1.9<sup>(−14.4)</sup></td><td>76.9<sup>(−9.8)</sup></td>
  </tr>
  <tr>
    <td>AI-Sweden/Llama-3-8B</td><td>36.7</td><td>73.3</td><td>30.3</td><td>71.4</td><td>58.5</td><td>94.3</td><td>41.9</td><td>91.2</td>
  </tr>
  <tr>
    <td>AI-Sweden/Llama-3-8B-IT</td><td>24.5<sup>(−12.2)</sup></td><td>73.3<sup>(+0.0)</sup></td><td>22.2<sup>(−8.1)</sup></td><td>72.7<sup>(+14.3)</sup></td><td>6.2<sup>(−52.3)</sup></td><td>80.3<sup>(−14.0)</sup></td><td>1.2<sup>(−40.7)</sup></td><td>68.0<sup>(−23.2)</sup></td>
  </tr>
  <tr>
    <td>Meta/Llama-3-8B</td><td>37.2</td><td>73.8</td><td>29.6</td><td>71.5</td><td>47.8</td><td>92.5</td><td>34.5</td><td>89.7</td>
  </tr>
  <tr>
    <td>Meta/Llama-3-8B-IT</td><td>30.1<sup>(−7.1)</sup></td><td>75.2<sup>(+1.4)</sup></td><td>26.1<sup>(−3.5)</sup></td><td>73.1<sup>(+1.6)</sup></td><td>30.1<sup>(−17.7)</sup></td><td>87.6<sup>(+4.9)</sup></td><td>3.2<sup>(−32.3)</sup></td><td>67.7<sup>(−22.0)</sup></td>
  </tr>
  <tr>
    <td>Mistral-Nemo-12B</td><td>34.0</td><td>72.5</td><td>27.8</td><td>69.2</td><td>49.5</td><td>92.9</td><td>35.7</td><td>90.1</td>
  </tr>
  <tr>
    <td>Mistral-Nemo-12B-IT</td><td>41.1<sup>(+7.1)</sup></td><td>76.1<sup>(+3.8)</sup></td><td>36.8<sup>(+9.0)</sup></td><td>75.0<sup>(+4.8)</sup></td><td>7.4<sup>(−41.1)</sup></td><td>82.4<sup>(+6.9)</sup></td><td>2.4<sup>(−33.3)</sup></td><td>72.4<sup>(−27.7)</sup></td>
  </tr>
</table>

Table 11: Performance scores of the pretrained-only and instruction-finetuned Norwegian LMs on our text summarization and machine translation tasks. The LMs are evaluated in (i) a zero-shot regime on NorSumm and (ii) a 16-shot regime on Tatoeba. En=English; BM=Norwegian Bokmål; NN=Norwegian Nynorsk. Cold-colored cells represent cases where an instruction-tuned version improves performance compared to the base LM, while warm-colored cells indicate cases where it decreases. The best score is in bold, the second best is underlined.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Text Summarization</th>
    <th colspan="2">Text Rewriting</th>
  </tr>
  <tr>
    <th>NorSummarize-Instruct</th>
    <th>BERTScore</th>
    <th>NorRewrite-Instruct</th>
    <th>BERTScore</th>
  </tr>
  <tr>
    <td>chrF</td>
    <td>BERTScore</td>
    <td>chrF</td>
    <td>BERTScore</td>
  </tr>
  <tr>
    <td>NB-GPT-6B</td><td>23.8</td><td>57.0</td><td>19.5</td><td>56.1</td>
  </tr>
  <tr>
    <td>GPT-SW3-6.7B</td><td>20.7</td><td>54.4</td><td>18.2</td><td>48.9</td>
  </tr>
  <tr>
    <td>NorwAI-Mistral-7B</td><td>22.2</td><td>54.7</td><td>20.4</td><td>53.6</td>
  </tr>
  <tr>
    <td>NorwAI-Llama2-7B</td><td>21.6</td><td>53.7</td><td>21.1</td><td>54.3</td>
  </tr>
  <tr>
    <td>NorBLOOM-7B-warm</td><td>9.0</td><td>24.0</td><td>5.2</td><td>17.2</td>
  </tr>
  <tr>
    <td>NorMistral-7B-scratch</td><td>8.5</td><td>24.0</td><td>7.2</td><td>20.0</td>
  </tr>
  <tr>
    <td>Viking-7B</td><td>21.4</td><td>55.7</td><td>21.8</td><td>55.7</td>
  </tr>
  <tr>
    <td>NorMistral-11B</td><td>27.2</td><td>61.4</td><td>25.7</td><td>71.0</td>
  </tr>
  <tr>
    <td>Viking-13B</td><td>21.1</td><td>55.4</td><td>22.8</td><td>56.0</td>
  </tr>
  <tr>
    <td>NorMistral-7B-warm</td><td>6.7</td><td>22.1</td><td>6.7</td><td>23.1</td>
  </tr>
  <tr>
    <td>NorMistral-7B-warm-IT</td><td>41.4<sup>(+34.7)</sup></td><td>71.2<sup>(+49.1)</sup></td><td>41.2<sup>(+34.5)</sup></td><td>70.7<sup>(+47.6)</sup></td>
  </tr>
  <tr>
    <td>Mistral-7B</td><td>5.7</td><td>15.9</td><td>6.0</td><td>18.8</td>
  </tr>
  <tr>
    <td>Mistral-7B-IT</td><td>31.7<sup>(+26.0)</sup></td><td>70.3<sup>(+54.4)</sup></td><td>29.5<sup>(+23.5)</sup></td><td>70.0<sup>(+57.2)</sup></td>
  </tr>
  <tr>
    <td>AI-Sweden/Llama-3-8B</td><td>21.2</td><td>54.4</td><td>21.9</td><td>55.0</td>
  </tr>
  <tr>
    <td>AI-Sweden/Llama-3-8B-IT</td><td>32.3<sup>(+11.1)</sup></td><td>68.8<sup>(+14.4)</sup></td><td>30.3<sup>(+8.4)</sup></td><td>68.8<sup>(+13.8)</sup></td>
  </tr>
  <tr>
    <td>Meta/Llama-3-8B</td><td>21.8</td><td>55.4</td><td>20.4</td><td>52.0</td>
  </tr>
  <tr>
    <td>Meta/Llama-3-8B-IT</td><td>35.4<sup>(+13.6)</sup></td><td>71.9<sup>(+16.5)</sup></td><td>29.9<sup>(+9.5)</sup></td><td>68.5<sup>(+16.5)</sup></td>
  </tr>
  <tr>
    <td>Mistral-Nemo-12B</td><td>18.7</td><td>47.3</td><td>18.1</td><td>49.9</td>
  </tr>
  <tr>
    <td>Mistral-Nemo-12B-IT</td><td>39.9<sup>(+21.2)</sup></td><td>72.2<sup>(+24.9)</sup></td><td>38.9<sup>(+20.8)</sup></td><td>71.8<sup>(+21.9)</sup></td>
  </tr>
</table>

Table 12: Performance scores of the pretrained-only and instruction-finetuned Norwegian LMs on our instruction-style text summarization and rewriting tasks. The LMs are evaluated in a zero-shot regime on NorSummarize-Instruct and NorRewrite-Instruct. Cold-colored cells represent cases where an instruction-tuned version improves performance compared to the base LM, while warm-colored cells indicate cases where it decreases. The best score is in bold, the second best is underlined.
F  Automatic Evaluation of Instruction-tuned LMs via LLM-as-a-judge

We evaluate the instruction-following abilities of the instruction-tuned LMs prompted for creative rewriting and summarization. Such generative tasks are difficult to evaluate even with access to the gold standard references. We use the LLM-as-a-judge approach, which involves a side-by-side comparison of LMs’ responses using an external judge LM. While judge models suffer from various biases (Chen et al., 2024; Wang et al., 2024; Li et al., 2025), they correlate with human judgments better than traditional language generation performance metrics (Sai et al., 2022; Zheng et al., 2023).

Expected win-rate scores  Given an instruction \( i \), two outputs \( o_A \) and \( o_B \) from LMs \( A \) and \( B \), and a human reference \( o_R \), the judge model \( \theta \) computes a score function:

\[
s_{\theta}(i, o_A, o_B, o_R) = \begin{cases}
1, & \text{if } o_A \succ_{\theta} o_B \quad (\theta \text{ prefers } o_A \text{ over } o_B) \\
0, & \text{if } o_A \prec_{\theta} o_B \\
1/2, & \text{otherwise.}
\end{cases}
\]

Using this, we can compute the expected win-rate of LM \( A \) over LM \( B \) as the expected value of the score function over a distribution \( \mathcal{D} \) of prompts and human references:

\[
\text{win\_rate}_{\theta}(A, B) = \frac{1}{2} \left( 1 + \mathbb{E}_{i,o \sim \mathcal{D}} s_{\theta}(i, A(i), B(i), o) - \mathbb{E}_{i,o \sim \mathcal{D}} s_{\theta}(i, B(i), A(i), o) \right)
\]

where the second symmetric term prevents position bias (Wang et al., 2024) from influencing the results.

Judge Model  Unlike Lyu et al. (2024), we use simple chain-of-thought prompting by asking the model to first describe the qualities of each response before giving the final verdict – this is done to further improve the evaluation accuracy (Wei et al., 2022). The judge is instructed to end its output by either generating “A” (for preference of response \( A \)), “B” (for preference of response \( B \)), or “tie” (for cases when both responses are either equally good or bad). We then parse the output and assign a score value according to Equation (1). A response pair is skipped in case of an incorrectly formatted judgment, which has not empirically occurred in our experiments.

F.1  Evaluating LLM-as-a-judge

<table>
  <tr>
    <th></th>
    <th>A</th>
    <th>Tie</th>
    <th>B</th>
  </tr>
  <tr>
    <th>A</th>
    <td>35.42%</td>
    <td>8.33%</td>
    <td>12.50%</td>
  </tr>
  <tr>
    <th>Tie</th>
    <td>8.33%</td>
    <td>4.17%</td>
    <td>4.17%</td>
  </tr>
  <tr>
    <th>B</th>
    <td>10.42%</td>
    <td>6.25%</td>
    <td>10.42%</td>
  </tr>
</table>

Table 13: Agreement between the judge model and humans. We compute the rates using prompt template from Appendix F.2 without human references.

We analyze the agreement rate between the judge model and humans, as well as self-preference bias on an BM subset of Primeape,\footnote{An ongoing work on multilingual human preference prediction and explanation at the moment of writing a camera-ready version of this paper; the data is provided by the dataset creators in response to our request. To be available at github.com/Toloka/primeape.} and explore language and position biases on NorRewrite-Instruct and NorSummarize-Instruct. The Primeape subset consists of 48 pairwise comparisons between meta-llama/Llama-3.3-70B-Instruct, google/gemma-2-27b-it, mistralai/Mistral-Small-Instruct-2409, Claude 3.7 Sonnet, and GPT-4o (Hurst et al., 2024), evaluated over 19 manually translated instructions from AlpacaEval (Dubois et al., 2023).
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="5">NorRewrite-Instruct</th>
    <th colspan="5">NorSummarize-Instruct</th>
  </tr>
  <tr>
    <th>NOB</th>
    <th>NNO</th>
    <th>SWE</th>
    <th>DAN</th>
    <th>ENG</th>
    <th>NOB</th>
    <th>NNO</th>
    <th>SWE</th>
    <th>DAN</th>
    <th>ENG</th>
  </tr>
  <tr>
    <td>Instructions</td>
    <td><b>99.3%</b></td>
    <td>0.7%</td>
    <td>0.0%</td>
    <td>0.0%</td>
    <td>0.0%</td>
    <td><b>96.4%</b></td>
    <td>3.6%</td>
    <td>0.0%</td>
    <td>0.0%</td>
    <td>0.0%</td>
  </tr>
  <tr>
    <td>NorMistral-7B-warm-IT</td>
    <td><b>98.6%</b></td>
    <td>0.7%</td>
    <td>0.0%</td>
    <td>0.0%</td>
    <td>0.7%</td>
    <td><b>99.0%</b></td>
    <td>0.5%</td>
    <td>0.0%</td>
    <td>0.0%</td>
    <td>0.5%</td>
  </tr>
  <tr>
    <td>Mistral-Nemo-12B-IT</td>
    <td><b>87.5%</b></td>
    <td>0.7%</td>
    <td>0.0%</td>
    <td>1.4%</td>
    <td>9.7%</td>
    <td><b>77.2%</b></td>
    <td>0.0%</td>
    <td>0.0%</td>
    <td>0.5%</td>
    <td>21.8%</td>
  </tr>
  <tr>
    <td>Mistral-7B-IT</td>
    <td>29.9%</td>
    <td>0.0%</td>
    <td>0.0%</td>
    <td>6.9%</td>
    <td><b>63.2%</b></td>
    <td>35.5%</td>
    <td>0.0%</td>
    <td>0.0%</td>
    <td>4.6%</td>
    <td><b>59.4%</b></td>
  </tr>
  <tr>
    <td>Meta/Llama-3-8B-IT</td>
    <td>34.0%</td>
    <td>0.0%</td>
    <td>0.0%</td>
    <td>0.0%</td>
    <td><b>66.0%</b></td>
    <td><b>49.7%</b></td>
    <td>0.0%</td>
    <td>0.0%</td>
    <td>0.5%</td>
    <td>49.2%</td>
  </tr>
  <tr>
    <td>AI-Sweden/Llama3-8B-IT</td>
    <td>0.0%</td>
    <td>0.0%</td>
    <td><b>100.0%</b></td>
    <td>0.0%</td>
    <td>0.0%</td>
    <td>0.0%</td>
    <td>0.0%</td>
    <td><b>100.0%</b></td>
    <td>0.0%</td>
    <td>0.0%</td>
  </tr>
</table>

Table 14: Language distribution in model responses on NorRewrite-Instruct and NorSummarize-Instruct. We show the percentages of instructions and responses in Norwegian Bokmål, Nynorsk, Swedish, Danish and English.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="5">NorRewrite-Instruct</th>
    <th colspan="5">NorSummarize-Instruct</th>
  </tr>
  <tr>
    <th>NorMistral-7B-warm-IT</th>
    <th>Mistral-Nemo-12B-IT</th>
    <th>Mistral-7B-IT</th>
    <th>Meta/Llama-3-8B-IT</th>
    <th>Average</th>
    <th>NorMistral-7B-warm-IT</th>
    <th>Mistral-Nemo-12B-IT</th>
    <th>Mistral-7B-IT</th>
    <th>Meta/Llama-3-8B-IT</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>NorMistral-7B-warm-IT</td>
    <td>—</td>
    <td>38.9</td>
    <td>79.2</td>
    <td>24.7</td>
    <td>87.5</td>
    <td>57.6%</td>
    <td>—</td>
    <td>33.8</td>
    <td>59.4</td>
    <td>18.1</td>
    <td>83.2</td>
    <td>48.6%</td>
  </tr>
  <tr>
    <td>Mistral-Nemo-12B-IT</td>
    <td>61.1</td>
    <td>—</td>
    <td>84.8</td>
    <td>43.8</td>
    <td>91.4</td>
    <td>70.3%</td>
    <td>66.2</td>
    <td>—</td>
    <td>77.3</td>
    <td>38.3</td>
    <td>91.1</td>
    <td>68.2%</td>
  </tr>
  <tr>
    <td>Mistral-7B-IT</td>
    <td>20.8</td>
    <td>15.2</td>
    <td>—</td>
    <td>4.9</td>
    <td>60.7</td>
    <td>25.4%</td>
    <td>40.6</td>
    <td>22.7</td>
    <td>—</td>
    <td>15.1</td>
    <td>67.5</td>
    <td>36.5%</td>
  </tr>
  <tr>
    <td>Meta/Llama-3-8B-IT</td>
    <td>75.3</td>
    <td>56.2</td>
    <td>95.1</td>
    <td>—</td>
    <td>92.7</td>
    <td>79.8%</td>
    <td>81.9</td>
    <td>61.7</td>
    <td>84.9</td>
    <td>—</td>
    <td>97.9</td>
    <td>81.6%</td>
  </tr>
  <tr>
    <td>AI-Sweden/Llama-3-8B-IT</td>
    <td>12.5</td>
    <td>8.6</td>
    <td>39.3</td>
    <td>7.3</td>
    <td>—</td>
    <td>16.9%</td>
    <td>16.8</td>
    <td>8.9</td>
    <td>32.5</td>
    <td>2.1</td>
    <td>—</td>
    <td>15.0%</td>
  </tr>
</table>

Table 15: Instruction-finetuned LMs’ win-rates (%) when evaluating for the language bias. Cold-colored cells indicate high win-rate, while warm-colored cells indicate low win-rate.

Agreement with Human Annotators The agreement matrix is shown in Table 13. The results indicate that the judge model agrees with human annotators in 50-0% of the cases, compared to a random agreement rate of 33.3%. When ties are excluded, the agreement rate increases to 66.7%, with the corresponding random baseline of 50.0%. The results and their interpretation can be affected by the limited sample size.

Self-preference Bias We evaluate the self-preference bias (Panickssery et al., 2024) of meta-llama/Llama-3.3-70B-Instruct to analyze how often the judge model favors its own outputs. Among 32 pairwise comparisons, the judge’s verdicts result in 8 wins, 8 ties, and 16 losses. These results suggest that the judge model exhibits relatively low self-preference bias and behaves fairly as an evaluator.

Language Bias Since we evaluate the Norwegian capabilities of LMs responding to Norwegian instructions, only responses written in Norwegian should be the preferred ones. We use GlotLID (Kargaran et al., 2023) to analyze the language distribution in the instructions as well as in the model responses (see Table 14). Surprisingly, only NorMistral-7B-warm-IT consistently answers in Norwegian. Other models often switch either to English or related Scandinavian languages. To better understand the effect of requiring the responses to be in Norwegian, we modify our LLM-as-a-judge prompt template from Appendix F.2 by explicitly instructing the judge to be invariant to the language of the responses. This allows us to measure the Norwegian instruction following capabilities rather than the quality of producing
Norwegian. Table 15 shows that the LM ranking has changed, with Meta/Llama-3-8B-IT becoming the most capable instruction-following LM. Conversely, NorMistral-7B-warm-IT has the expected win-rate of 58% and 48%, which suggests that its high rate in the main experiment is more due to its capabilities to consistently producing Norwegian.

Position Bias  Position bias is a common bias within the LLM-as-a-judge paradigm, where a judge model prefers a response based on its position regardless of the content (Wang et al., 2024). While we mitigate this bias by evaluating each response pair twice with switched positions as shown in Equation (2), we observe a minor preference for the second position. Our judge prefers the first response 416× and the second one 538× on NorRewrite-Instruct; on NorSummarize-Instruct, the bias is less apparent – with 1 100 and 1 156 position preferences. In case of Primeape, we observe self-agreement rates of 45.83% with ties and 64.52% without ties. Overall, position bias appears to have an insignificant impact on NorRewrite-Instruct and NorSummarize-Instruct, whereas its effect is more pronounced on Primeape.

F.2 Prompt Template for LLM-as-a-judge

We adapt the HREF prompt template provided in Lyu et al. (2024) by localizing it to Norwegian and specifying that a Norwegian response should always be preferred over a non-Norwegian one.

System prompt:

You are a helpful assistant that helps us rate a Norwegian AI model’s responses to instructions.

User prompt:

Decide which response from the Norwegian AI system following the instruction is better, considering the following questions:
1. Most importantly, the AI systems should always respond in Norwegian. If a response is not in Norwegian, then you should consider it incorrect --- such a response should always be rated lower than any (even incorrect) response in Norwegian.
2. Does the response precisely follow the instruction? For example, a response that includes unrelated information or does not fulfill the task is not precisely following the instruction. Compare each response with the provided human response to decide if a response faithfully follows the instruction, especially when the instruction asks for expected word count or format.
3. Is the response helpful? For example, if the instruction asks for a recipe for healthy food, and the response is a useful recipe, then you can consider it helpful.
4. Is the language of the response natural? For example, AI responses are often verbose or repetitive, which is not natural. Compare with the provided human response to decide whether a response is natural.
5. Is the response factual/accurate? AI responses often make up new information. For example, if the response claims that Jens Stoltenberg is the current prime minister of Norway, then you should consider it inaccurate. Compare with the provided human response to verify whether a response is factual and accurate, especially with numbers.
6. Based on your aesthetics, which one do you prefer? For example, you might prefer one poem over another poem.

Select the response A or B that you prefer, or select tie if the two responses are similarly good or bad. Note that the responses can be truncated (don’t consider that as a mistake).

Here are three examples:

# Example 1:

## Instruction:
Omformuler følgende spørsmål: "Hva er hovedstaden i Frankrike?"

## Response A:
Hovedstaden i Frankrike er Paris.

## Response B:
Kan du fortelle meg navnet på byen som fungerer som hovedstaden i Frankrike?

## Human Response:
Hva heter Frankrikes hovedstad?

In this example, B paraphrases the question as asked by the instruction. In contrast, A does not follow instruction as it answers the question instead. Human Response also paraphrases the question, just in a slightly different way. To sum up, B is the best response because it follows the instruction.

## Which is best, A, B, or tie?
B

# Example 2:

## Instruction:
Bytt ut det første verbet med et synonym:
Jeg elsker å surf
## Response A:
Jeg hater å surfe

## Response B:
I like to surf

## Human Response:
Jeg liker å surfe

Response A tries to follow the instruction as it changes the first verb of the sentence, but it uses an antonym instead of a synonym. The response B might be correct, but it is written in English, not Norwegian, and non-Norwegian responses should always be rated as worse. Human Response changes the first verb, "elsker" (love), into its synonym, "liker" (like), as asked by the instruction. In conclusion, A is better than B because it is written in Norwegian.

## Which is best, A, B, or tie?
A

# Example 3:

## Instruction:
Bytt ut det første verbet med et synonym:
Jeg elsker å surfe

## Response A:
Jeg hater å surfe

## Response B:
Jeg liker ikke å surfe

## Human Response:
Jeg liker å surfe

In this example, neither output is correct and the responses are very similar. Human Response changes the first verb into its synonym, as asked by the instruction. To conclude, both A and B are equally incorrect, so the answer is tie.

## Which is best, A, B, or tie?
tie

Now here is the real task, first describe the qualities of each response and then end your message by writing "# Which is best, A, B, or tie?" and selecting among: A, B, or tie.

# Task:

## Instruction:
{{instruction}}

## Response A:
{{output_1}}

## Response B:
{{output_2}}

## Human Response:
{{output_human}}

Reviewer paper summaries: The paper introduces NorEval, a comprehensive benchmark designed for evaluating Norwegian generative language models. NorEval consists of various datasets (including five newly human- created), covering diverse language understanding and generation tasks across both Norwegian written standards—Bokmål and Nynorsk. The benchmark evaluates 19 open-source models, incorporates diverse prompts via the LM Evaluation Harness framework, and establishes human performance baselines. Key findings highlight the varying strengths of Norwegian language models across different tasks, particularly in instruction-following scenarios, and reveal significant performance differences between Bokmål and Nynorsk.

The paper presents NorEval, an extensive evaluation suite designed for benchmarking Norwegian generative language models (LMs). The benchmark supports both official written standards in Norway - Bokmål and Nynorsk.
Bokmål is the primary written system while Nynorsk is used by 10%-15% of the population. While these systems are related they exhibit lexical and grammatical differences. All existing datasets for Norwegian has small or no support for Nynorsk. The work in this paper addresses this gap by intentionally increasing the representation of Nynorsk.


NorEval includes 24 high-quality human-created datasets spread across nine task categories:
- Sentiment analysis
- Norwegian language knowledge
- Norwegian-specific & world knowledge
- Reading comprehension, 
- Commonsense reasoning
- Machine translation
- Text summarization
- Instruction following
- Truthfulness

Overall 24 datasets are used - 19 gathered from existing peer-reviewed previously published sources and additional 5 datasets developed from scratch to enhance linguistic diversity and quality. 

The evaluation framework supports various scenarios by giving users the ability to play with 100+ dataset-specific prompts that cover diverse user instructions. 
For each dataset, the authors generated 4-6 different prompts - some translated from PromptSource (an open source pool of prompts) while others were written from scratch. Each prompt was adapted from Bokmål to Nynorsk.

The authors experiment with pre-trained, instruction-tuned Norwegian pretrained decoder-only LMs publicly available in Huggingface Transformers library.
Each model is evaluated in 3 k-shot (k = {0, 1, 16}) in-context learning settings.

Generation tasks are evaluated by a LLM-as-a-judge LLaMA-3.3-70B-Instruct in a pairwise win-rate comparison between outputs (the judge is given the input and 2 outputs generated by 2 different models and decides which one looks better).
Final scores across the entire benchmark are computed using 3 different scoring aggregation methods
- Multi-prompt aggregation - using high score for each model across each dataset prompts
- Average Normalized Score - rescaling model scores across 9 task categories by normalizing between random baseline and max possible score.
- Borda's Count - assign scores to each model based on its rank-position in each task as a weighted sum.
The benchmark is integrated into an existing previously released framework namely - LM Evaluation Harness.
In addition human baselines for some datasets were created by sampling 50 examples and giving them to 3 different annotators.


The results reveal high performance for tasks like sentiment analysis and machine translation but highlight challenges in tasks requiring Norwegian language knowledge and instruction following.
Summary of strengths: - The benchmark is the largest Norwegian-specific evaluation suite to date, offering diverse tasks covering both Bokmål and Nynorsk standards.
- Compared to previous benchmarks, NorEval uniquely includes language-specific knowledge tasks such as punctuation rule adherence and Norwegian idiom completion, along with instruction-following tasks like NorSummarize-Instruct and NorRewrite-Instruct.
- The paper is well-written, with clear descriptions of annotation guidelines and evaluation procedures.
- The authors perform extensive experiments across various evaluation settings and propose using the Borda count metric to rank models robustly, minimizing biases from task-specific scoring scales.

1. **Comprehensive Coverage:** NorEval includes 24 datasets covering a broad spectrum of tasks important for evaluating language models comprehensively.

2. **Human Baseline Establishment:** The introduction of human performance baselines provides a clear upper performance bound for LMs, enhancing the benchmarking framework.

3. **Attention to Linguistic Diversity:** Inclusion of both Bokmål and Nynorsk datasets aims to improve representation, addressing the underrepresentation of Nynorsk in previous works.

4. **Reproducibility and Accessibility:** The integration of datasets into the LM Evaluation Harness facilitates flexible and reproducible evaluation, encouraging transparency and ease of use.

5. **Significant Contributions to Resource-lean Languages:** The benchmark addresses a critical gap in resources for Norwegian, a low-resource language, potentially encouraging further developments in NLP for Nordic languages.
Summary of weaknesses: - Random sampling in few-shot evaluation setups could introduce variability in reported performance.
- Although the authors created four to six prompts per dataset, the differences among prompts for the same task are not sufficiently clear in the main manuscript and require reviewing supplementary sections.
- The presentation of delta scores comparing Bokmål and Nynorsk is ambiguous. It is unclear whether scores represent BM-NN or NN-BM differences.
- The evaluation of generative tasks predominantly relies on automatic metrics and LLM-as-judge evaluations, lacking thorough human validation, a limitation acknowledged by the authors.
 - A human baseline is provided for this benchmark, considered by the authors as the upper performance bound for evaluating Norwegian language models, although this might not necessarily represent a true upper limit.

1. **Data Contamination Concerns:** The paper acknowledges potential test data contamination from open sources but does not provide a detailed analysis or estimation of its impact.

2. **Language Model Selection:** The selection of models seems focused on publicly available options, possibly excluding other competitive proprietary models that could provide additional insights.

3. **Limited Discussion on Unexpected Results:** The paper does not extensively discuss unexpected results, such as in tasks requiring Norwegian language knowledge, or outline potential reasons or hypotheses for the findings.
Comments, suggestions and typos: - The authors might conduct multiple runs for few-shot evaluations to reduce variability and enhance the robustness of results.
- Refine aggregation methods or provide detailed, task-level scores separately to improve interpretability.
- LLM-as-a-judge assessments might be cross-validated with human evaluations for at least a random set, to ensure the reliability of LLMs as judges in this scenario as well.

- Consider including competitive proprietary or cutting-edge Norwegian LMs to offer a more rounded comparison and to potentially identify state-of-the-art solutions.

- Consider discussing reasons behind unexpected performance outcomes on different task types to offer readers deeper insights into the challenges and opportunities present in the benchmarking results.
Overall assessment scores: 3

3.5
Metareview score: 3.5
Metareview: The paper introduces a comprehensive benchmark designed for evaluating Norwegian generative language models, with both official written standards in Norway (Bokmål and Nynorsk) and tasks that necessitate language-specific knowledge. 19 open-source models are tested, with instruction-tuning.
Summary of reasons to publish: - The paper proposes a very comprehensive benchmark for Norwegian, it is clear and well-written.
- The paper increases linguistic diversity in NLP, as the inclusion of both Bokmål and Nynorsk datasets adresses the underrepresentation of Nynorsk in previous work.
- The benchmark includes language-specific knowledge tasks, such as punctuation rule adherence and Norwegian idiom completion, along with instruction-following tasks like NorSummarize-Instruct and NorRewrite-Instruct.
- The paper offers extensive experiments across various evaluation settings.