Title: NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegian
Benchmark: NLEBench
Summary: This paper introduces NLEBench, a benchmark designed to evaluate generative language models in Norwegian. It also presents NorGLM, a series of open-source generative language models trained specifically for Norwegian, and evaluates them on the benchmark.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Natural language generation in Norwegian

Phenomenon defined by authors: No
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Not defined
Task definition: Multi-task benchmark: instruction finetuning (NO-Alpaca), question answering (NO-BoolQ), natural language inference (NO-QNLI), paraphrase (NO-MRPC), open-domain conversation (NO-ConvAI2), summarization (NO-CNN/DailyMail), bias detection and toxicity detection (NO-CrowS-Pairs), multi-task learning (NO-Multi-QA-Sum)
Task item definition: Different per task: for instruction finetuning it's a prompt, for question answering a question, for NLI a pair of sentences, for paraphrase a sentence, for open-domain conversation a dialogue, for summarization an article, for bias detection and toxicity detection a pair of sentences, for multi-task learning an article 
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Expert-crafted task examples (e.g. hand-written examples), Modified from another benchmark (e.g. translation into another language)
Task ecology: Complete real task (e.g. providing medical advice to real people interactively), Partial real task (e.g. answering medical questions collected from real people), Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: More than 271.000.000 total samples
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), Distribution (perplexity, calibration, correlation), Toxicity Score, Entailment Score
Metric definition detail: Model access required for perplexity
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: Multi-task learning scores are divided based on whether the QA included a summary or not. Toxicity scores are provided by level (toxicity, severe toxicity, identity attack, insult, profanity, threat) and bias by type (race-color, socioeconomic, gender, disability, nationality, sexual orientation, physical appearance, religion, age, politics).
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Authors address the Face and Content aspects of the construct validity. They justify that most benchmarks are single-task and argue their multi-task design is more aligned with real-world usage, including culturally grounded instructions and human-generated summaries. Moreover, the authors explain their task design: tasks are drawn from a diverse set of abilities, some are manually created and the manual translation process is reviewed.
Taxonomy: NLP / 
Alternate taxonomy: Multilinguality
New taxonomy: General Capability / Natural Language Understanding
Procedural notes: Datasets and models available on Hugging Face: https://huggingface.co/NorGLM