Title: Superlim: A Swedish Language Understanding Evaluation Benchmark
Benchmark: Superlim
Summary: We present Superlim, a multi-task NLP bench- mark and analysis platform for evaluating Swedish language models, a counterpart to the English-language (Super)GLUE suite. From the set of experiments, it is quite challenging to the models.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: natural language understanding
Phenomenon defined by authors: Yes
Phenomenon definition: NLU includes a wide range of subtasks such as sentiment analysis, argumentation classification, grammatical error detection, semantic similarity, natural language inference, coreference resolution, word similarity and relatedness, analogy, synonym detection, and diagnostics for linguistic phenomena and gender bias.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The Superlim benchmark defines its tasks as a set of 15 NLU tasks for Swedish, covering text-level tasks (e.g., sentiment analysis, NLI, paraphrase detection), word-level tasks (e.g., similarity, analogy), and diagnostic tasks (e.g., gender bias detection, linguistic phenomenon inference).
Task item definition: A single item in a task dataset typically consists of text inputs (such as a sentence, sentence pair, or word pair) with the respective label or target output specific to the task—e.g., a sentiment score or a classification label.
Task source: Human exam questions (e.g. GRE questions), Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 36,118 (the range is from 109 examples to 18,593)
Dataset size notes: total (479,571 train) and (22,527 validation)
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragarph), predicted label
Metric definition: Krippendorff’s α
Metric aggregation: Simple Mean
Metric statistics: simple mean, std
Metric subscores: No
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: there is no link in the paper, but can find it online
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / Understanding
New taxonomy: General Capability / Natural Language Understanding