Superlim: A Swedish Language Understanding Evaluation Benchmark
Authors: Aleksandrs Berdicevskis
Gerlof Bouma
Robin Kurtz
Felix Morger
Joey Öhman
Yvonne Adesam
Lars Borin
Dana Dannélls
Markus Forsberg
Tim Isbister
Anna Lindahl
Martin Malmsten
Faton Rekathati
Magnus Sahlgren
Elena Volodina
Love Börjeson
Simon Hengchen
Nina Tahmasebi
Venue: EMNLP 2023 Main (2023)
Keywords: Swedish
benchmark
large language models
natural language understanding
transfer learning
evaluation

Abstract: We present Superlim, a multi-task NLP benchmark and analysis platform for evaluating Swedish language models, a counterpart to the English-language (Super)GLUE suite. We describe the dataset, the tasks, the leaderboard and report the baseline results yielded by a reference implementation. The tested models do not approach ceiling performance on any of the tasks, which suggests that Superlim is truly difficult, a desirable quality for a benchmark. We address methodological challenges, such as mitigating the Anglocentric bias when creating datasets for a less-resourced language; choosing the most appropriate measures; documenting the datasets and making the leaderboard convenient and transparent. We also highlight other potential usages of the dataset, such as, for instance, the evaluation of cross-lingual transfer learning.

Superlim:
A Swedish Language Understanding Evaluation Benchmark
Aleksandrs Berdicevskis' * Gerlof Bouma" Robin Kurtz² Felix Morger" Joey Öhman
gerloof bouma, fewie.oorgeereeeeeeeeeeeee morii.m.orgerrgu.... Beainnkurttkk.... se joey. ohman@embark-studdoon com
Yvonne Adesam" Lars Borin"
Dana Dannélls' Markus Forsberg" Tim Isbister
Anna Lindahl"
Martin Malmsten'
Faton Rekathati
Magnus Sahlgren
Elena Volodina"
Love Börjeson'
Simon Hengchen' Nina Tahmasebi
Spräkbanken Text, University of Gothenburg
KBLab, National Library of Sweden $ Embark Studios
AI Sweden 'ggannddnn.ai
4 University of Gothenburg
Abstract
standing. It contains 14 different datasets which cover 15 tasks ranging from word understanding tasks, such word analogy and word similarity, to text classification tasks, such as natural language inference and sentence acceptability. The release of this benchmark comes at a time when multiple pre-trained language models have been created for Swedish, and larger models are under construc- tion (ranging from Malmsten et al., 2020 to Ekgren et al., 2023). In this active research environment, a multi-task benchmark is crucial to gauge state-of- the-art performance and guide future development. For Swedish this is especially important, since lim- itations (financial, computational and related data availability) are more considerable than for English.
We present Superlim, multi-task NLP bench- mark and analysis platform for evaluating Swedish language models, a counterpart to the English-language (Super)GLUE suite. We de- scribe the dataset, the tasks, the leaderboard and report the baseline results yielded by a erence implementation. The tested models do not approach ceiling performance the tasks, which suggests that Superlim is truly difficult, desirable quality for benchmark. We address methodological challenges, such as mitigating the Anglocentric bias when creating datasets for less-resourced language; choos- ing the most appropriate measures; document- ing the datasets and making the leaderboard convenient and transparent. We also highlight other potential usages the dataset, such as, for instance, the evaluation of cross-lingual transfer learning.
any of
Creating benchmark for Swedish comes with its own unique set of practical challenges and methodological considerations. While there exist high-quality Swedish datasets that can be reused, not all them have been created with the task of evaluating large language models in mind. Here, the relatively close linguistic relationship to En- glish comes with possibilities of translating (or adapting) existing datasets. Such approaches do however potentially introduce linguistic and cul- tural Anglocentric bias into Swedish language benchmarking. Therefore, Superlim combines datasets of three kinds: a) preexisting ones, refor- matted and revised, b) translated or adapted from English, and c) created from seratch specifically for Superlim.
1 Introduction
In recent years, the move towards Transformer- based large pretrained language models has brought the need for comprehensive and standard- ized multi-task benchmarking in NLP into focus, Since these models, through their generic pre- training setup, can be fine-tuned on many different tasks, test suite which covers a range of differ- ent NLP domains is required to measure overall performance. For this reason, several different benchmarks have been proposed. For English, the most notable suites arguably the (SuPer)GLUE benchmark (Wang et al., 2018, 2019) for natu- ral language understanding and GEM (Gehrmann et al., 2021) 1 for natural language generation.
To facilitate progress tracking and sharing of results, models and datasets, we create an online leaderboard for Superlim, which will serve as a platform for submitting and publishing results, and
In this paper, we present Superlim, a multi-task benchmark for Swedish natural language under-

---

release reference implementations with baseline re- sults for collection of existing Swedish language models.
tion about the datasets and their creation, we refer to the documentation that accompanies Superlim, itself described Section 4. Table lists all the tasks in Superlim.
The rest of the paper looks as follows: In Sec- tion 2 we briefly cover related work. Section 3 describes the tasks included in Superlim and the corresponding datasets, and Section 4 their stan- dardized documentation. Section 5 motivates Su- perlim's main evaluation measure: KrippendorfI's a. The online design of the leaderboard the sub- ject of Section Sections 7 and 8 describe the reference implementations and the baseline results. Section 9 provides details on where to find Super- lim and Section 10 concludes the paper.
Most datasets are divided into predefined train- ing, development and test splits. All test sets are of gold-standard quality: they have been manually annotated from scratch or manually corrected after automatic processing. Some of the training and de- velopment sets may contain non-corrected data, for instance automatically translated data. Our ratio- nale here is that model can be trained on anything, as long as it yields high, robust, unbiased perfor- mance. Evaluation, however, must always occur on gold data to serve its purposc. An important caveat, however, is that those test sets that were automati- cally translated and then manually corrected may suffer from so-called post-editese (Toral, 2019). In all cases, the editing was carefully done by native speakers with background in NLP and/or linguis- tics, who were aware of this potential problem. We hope that these factors have at least mitigated it.
2 Related work
The already mentioned (Super)GLUE has inspired the creation of similar benchmarks for other lan- guages: CLUE (Xu et al., 2020) and CLiMP (Xi- ang et al., 2021) for Chinese, Russian Super- GLUE (Shavrina al., 2020), KLEJ (Rybak et al., 2020) and LEPISZCZE (Augustyniak et al., 2022) for Polish, ParsiNLU for Persian (Khashabi et al., 2021), IndoNLU for Indonesian (Wilie et al., 2020) and NorBench (Samuel et al., 2023) for Norwe- gian. Several suggestions to move to other evalua- tion upproaches have been voiced (Ethayarajh and Jurafsky, 2020; Ribeiro al., 2020; Kiela al., 2021). but the GLUE-like benchmarks still remain the default approach.
The word-level datasets were mostly created with non-contextuallized type level approaches in mind (for instance, static word vectors like those obtained from word2vec). These therefore do not contain large training splits, but small training sets that can be used to calibrate otherwise unsuper- vised methods, or for few-shot training generative models. The diagnostic datasets only have test splits. Here, the training and development splits of SweNLI are to be used.
Superlim is not the only benchmark for Swedish. Very recently, ScandEval has been released (Nielsen, 2023). This multilingual suite contains four tasks for Swedish. Some of the datasets used for the evaluation contain synthetic data (for in- stance, incorrect sentences created by serambling word order in the correct sentences), which is not necessarily problematic, but presumably less valu- able than natural data. Another benchmark for Swedish, OverLim (Kurtz, 2022), is entirely ma- chine translated from English.
3.1
Text-level tasks
3.1.1 Absabank-Imm
Absabank- -Im derived from the Swedish aspect- based sentiment (stance) analysis corpus described in Rouces et al. (2020). We selected the part anno- tated for author attitude regarding immigration in Sweden, quantified as real-valued scores between (very negative) and 5 (very positive). These scores are the arithmetic mean of integer scores from indi- vidual annotators, but since most paragraphs were labelled by only one annotator, most average values are also integer.
3 Tasks and datasets
We have three types of tasks: text level (Sec- tion 3.1), word level (Section 3.2) and diagnos- tie (Section 3.3). Describing the creation of all datasets in detail is beyond the scope of this pa- per. Instead focus on briefly describing the datasets, their tasks and any substantial changes that have been made to the original sources for the purposes of Superlim. For more detailed informa-
The data for this task comes from different sources: editorials and opinion pieces from the national Swedish newspapers Svenska daghladet and Aftonbladet, and forum posts from Flashback, a large Swedish discussion forum.

---

dataset (items)
Task
Name
Type
Data source(s)
Translated
train
dev
test
Text level
stance
Absabank-Ima Imm
Score
Newspapers and forum
3 3898
487
487
argumentation
Argument. sents
3-lab
Common Crawl
3450
750
065
acceptability
DaLAJ-GED
2-lab
Learner essays, lextbooks
35
4702
4.71
similarity
SweParaphrase
score
STS Benchmark
yes
5715
1499
1 378
summarization
SweDN
gen
Newspaper
yes
29 847
4 4229
3 745
QA
SweFAQ
select
Websites of authorities
781
110
109
entailment
SweNLI
3-lab
Various sources
yes
392 702
915
305
word meaning coreference
SweWiC
2-lab
SALDO, Eukalyptus, Wiktionary
4.666
500
0000
SweWinograd
2-lab
Winograd
yes
721
135
140
Word level relatedness
SuperSim rel ness
score
cx-999 WordSim353
partly partly
131
229
similarity
SuperSim similarity
score
SimLex-990 999, WordSim353
131
229
analogy
Swedish Analogy SweSAT
select
English Analogy, Wiktionary
partly
2 2045
18 593
synonymy
select
Scholastic Aptitude Test
83
739
Diagnostics
entailment
SweDiagnostics SweWinogender
3-lab
News, Wikipedia, ACL, Reddit Winogender
yes
104
entailment
3-lab
yes
624
Table 1: Overview of the tasks in Superlim. Task type is one of: binary labelling, ternary labelling, scoring (regression), answer selection (multiple choice, pairing) or text generation. dataset counts as translated when the training and test items translated (manually automatically) from its English counterpart.
3.1.2 Argumentation sentences
3.1.4 SweParaphrase
Argumentation sentences, the task is to catego- rize sentences into for,,ent.or, argument against, unrelated to a given topic. The topics include abortion, death penalty, nuclear power, marijuana legalization, minimum wage and cloning. The data for this task was machine translated from the ar- gument unit recognition and classification dataset described in Trautmann et al. (2020) and manually corrected by a native speaker of Swedish.
The SweParaphrase dataset consists of sentence pairs, and the task is to estimate their semantic similarity, that is, to what extent they express the same state of affairs to 5 (similar). The gold standard annotation is arithmetic mean of multiple judgements.
scale from 0 (dissimilar)
The dataset is based on the Semantic Textual Similarity Benchmark (STS-B) (Cer et al.. 2017), which was first automatically translated to Swedish using the Google Translate API (Isbister al.. 2021) and then manually corrected by native speaker. The similarity values taken from STS-B without any adjustment.
3.1.3 DaLAJ-GED
For DaLAI-GED, sentences have be classified being grammatically correct or not. Incorrect datapoints were collected from the Swedish learner corpus SwELL (Volodina et al., 2019). Sentences containing errors from this material were manipu- lated to contain only one error per datapoint a sen- tence with multiple mistakes can thus give rise to multiple datapoints. Correct datapoints were sam- pled from the same material and from the course book corpus COCTAILL (Volodina et al., 2014).
3.1.5 SweDN
SweDN (Monsen and Jönsson, 2021) a summa- rization task, the only sequence generation task included in Superlim. This dataset collects almost 40 thousand articles from the Swedish newspaper Dagens Nyheter. The lead paragraph in each article serves as its ground truth summary.
3.1.6 SweFAQ
The DaLAI-GED materials contain additional annotation, such the type and location of error, its recommended correction, and information about the sentence source. This additional annotation is not currently used in the task itself, but we consider it valuable for error analysis, and it is a potential basis for future tasks using this material.
The dataset for SweFAQ consists of answers frequently asked questions taken from websites of nine Swedish authorities, such as the Social Insur- ance Agency and the Swedish Tax Agency. The dataset contains 976 question-anse answer pairs that fall into 100 different categories, for instance COVID-

---

19 vaccination or parental benefits. Each datapoint in the dataset is a question and a list of all answers from the same category a randomized order. The task is to select the matching answer, by giving its index. The length of the list of question-answer pairs varies across categories.
Each datapoint contains fragment in which pro- noun and possible antecedent are highlighted. The system has to predict whether they are corefer- ent or not.
S SweWinograd was created by manual translation of the Winograd schema challenge (Levesque al., 2012), as included in SuperGLUE. Note that the Winograd schema challenge also present in the original GLUE benchmark, but as an entailment task.
3.1.7 SweNLI
SweNLI natural language inference task: for two sentences, a premise and a hypothesis, predict the relation between the two (neutral, contradic- tion or entailment). The SweNLJ dataset is derived from two existing resources. The test split was con- structed from a grammar- automatic Swedish translation of the FraCaS test suite (Cooper et al., 1996; j.junlöff and Siverbo, 2012). For Superlim, the Swedish FraCaS suite was extensively man- ually revised and culturally adapted Swedish real-world facts.
3.2 Word-level tasks
For each of the word-level tasks described below, we created small training splits by randomly select- ing 10% of the combined data, meant to enable few shot learning or calibration of existing models.
3.2.1 SuperSim relatedness and SuperSim similarity
For the training and development splits, a ma- chine translated version of the Multi-genre Natural Language Inference (Williams et al., 2018) dataset is used. The original data was collected from ten different texts ranging from transcribed telephone calls to magazine articles. We translated the dataset Swedish using the OPUS-MT machine transla- tion framework (Tiedemann and Thottingal, 2020).
Superlim includes two tasks based on the the Su- perSim dataset (Hengchen and Tahmasebi, 2021). SuperSim contains word pairs annotated for seman- tic similarity and for semantic relatedness, both with scores between 0 (low degree) and 10 (high degree). Semantic similarity is the extent to which two concepts share semantic properties, semantic relatedness refers to more general association be- tween the concepts. For instance, cup and coffee related, but not similar. SuperSim relatedness and SuperSim similarity concern scoring pairs for relatedness and similarity, respectively.
3.1.8 SweWiC
SweWiC is modelled after the word-in-context task deseribed in Pilehvar and Camacho-Collados (2019). A polysemous target word is provided in two contexts. The system's 's task is determine whether the word is used with the same meaning or not.
SuperSim's word pairs were translated from SimL (Hill et al., 2015) and WordSim-353 (Finkelstein et al., 2002) and then labelled by five native speakers of Swedish. The ground truth is the arithmetic mean of the labels, but the individual labels are preserved, too.
The development and test splits use example sen- tences collected for the Swedish lexical resource SALDO (Borin et al., 2013), and sentences taken from the Eukalyptus corpus (Adesam et al., 2015), which has word-sense annotation based on SALDO. Training data was constructed from Swedish Wik- tionary. Since the word-in-context task doesn't 't directly rely on the sense inventory itself, but only asks whether two instances use the same mean- ing or not, differences between Wiktionary and SALDO in terms of sense inventory are acceptable (see also Pilehvar and Camacho-Collados, 2019).
3.2.2 Swedish Analogy
The Swedish Analogy dataset (Adewumi al., 2022) consists analogies of the form "Stockholm is to Sverige ( ('Sweden') as Berlin is to Tyskland ('Germany'): ) The dataset contains different cat- egories of analogy, of semantic (real world facts, as above, or lexical semantic relations) or of mor- phological nature. The dataset is partly based on Mikolov et al. (2013) for English, but was en- riched with additional categories and translated to Swedish using automatic tools and Wiktionary matches, after which it was proof-read by native speakers.
3.1.9 SweWinograd
Winograa a coreference resolution task, cast as a binary mention-paie classification problem.
See https://sv. wiktionary.

---

In Swedish Analogy, analogies have to be com- pleted by predicting the fourth element given the first three. The set of items from which the model may select an answer is not limited by the task, but only by the vocabulary of the model itself.
Items in SweWinggender consist of a premise, short text containing a pronoun that allows only one reasonable interpretation of this pronoun, and an hypothesis that spells out some interpretation of this pronoun. For correctly resolved pronouns, there is an entailment relation between the two sentences. Each rremise-hypothest pair occurs three times in the dataset, each time with different pronoun: han 'he', hon 'she', hen "(singular) they'. For a model that does not display any gender bias, its performance on the inference task should not correlate with the choice of pronoun.
3.2.3 SWeSAT
SweSAT was created in the context of Super- lim, and collected from the synonyms part of the Swedish Scholastic Aptitude Test (Högskoleprovet) for the years 2006-2021. The task is, given a word or an expression, to select the best synonym from list of five candidates.
Documentation
3.3 Diagnostic tasks
The importance of documenting datasets cannot be overestimated. Still this step often overlooked in the NLP field. Since some parts of Superlim are revised versions pre-existing datasets, is partic- ularly important to be explicit about what changes were made. We therefore devised a documentation sheet: a template inspired by Gebru et al.'s (2021) "I "Datasheets" and Pushkarna al.'s 's (2022) "Data cards". Our template, however, more compact than the former and specifically adapted the Superlim context. Ducumentation sheets consist six sections:
Superlim comes with two diagnostic tasks, which are both cast as inference tasks. Participating sys- tems should use the training materials supplied for SweNLI.
3.3.1 SweDiagnostics
SweDiagnostics is a is manually translated ver- sion of the GLUE diagnostic dataset (Wang et al., 2018). The dataset is used for diagnosing sys- tem's ability to handle specific linguistic phenom- ena. The diagnostic task is cast as inference task, where the premise and hypothesis are identical but for a minimal manipulation, like the insertion of negation. The entailment relationship (entailment, neutral, or contradiction) relies on the linguistic phenomenon targeted. There are 33 fine-grained phenomena across four coarse-grained categories argument structure,
Identifying information: basic information about the dataset and its creators;
Usage: why the dataset was created and how it can be used;
3. Data: the description of the contents (includ- ing basic statistics, format, data source, collec- tion method, inter-annotator agreement etc.);
(lexical semanties, logic and common sense). Translating the English dataset, we made sure that the targeted linguis- tic phenomena are present even the resulting Swedish sentences and that the sentences are id- iomatic Swedish.
4. Ethics and caveats: ethical considerations, po- tential pitfalls, discouraged usage;
5. About documentation: information about the documentation and dataset versions.
In SuperGLUE, the same diagnostic dataset was used, but formulated as binary classification prob- lem. In Superlim, we have chosen to keep the original ternary version.
6. Other information, including a bibliography.
Each Superlim task is accompanied by a documen- tation sheet. The compactness of the template is intentional: improves the chances that documen- tation will actually be written, maintained and read. An example can be found Appendix A.
3.4 SweWinogender
SweWinogende is an inference task designed to diagnose gender bias in models. It is a reformula- tion of the pronoun resolution test set of Hansson al. (2021), which, in turn, was manually trans- lated from/inspired by Winogender (Rudinger et al., 2018). We mirror SuperGLUUE. in this choice, since it has an inference task reformulation of the English Winogender.
Evaluation measures
A benchmark containing mix of task types will almost by necessity involve wide range of eval- uation methods and measures. The appropriate choices depend on the tasks themselves and which

---

insights we would like to get into our models. Good research practice tells us evaluate models in those ways that teach us most about them. How- ever, in a benchmark "competition" setting, were we would like to summarize a model's performance with one final score, this leads to a problem. In GLUE, this is dealt with by providing average over very different beasts like accuracy, Matthews correlation coefficient, f-score, and Spearman's 's cor- relation coefficient (that have different ranges and different intepretations). In the context of the Su- perlim benchmark, we explore an alternative: the use (where possible) of single family of measures, Krippendorf''s a (Krippendorff, 2018).
Leaderboard
The benchmarking results collected at the Su- perlim leaderboard, where both development and test set results are made available. In addition to the evaluation outcomes on the different tasks, the leaderboard also lists information about a partici- pating model's 's type and size (in number of param- eters), and the size of the dataset used to pre-tram pre-train the model.
The Superlim project strives for transparency and openness, not just with regards to our datasets but also to the contents of the leaderboard. As part of this, we link participating models' predictions on development and test sets as well as scripts and configuration details used to produce those results. This facilitates replication of results and confirma- tion of their validity. There is an additional point with this level of transparency: The gold labels of the Superlim test sets are not hidden. This makes evaluation easier, but it also makes cheating (in- tentional or accidental, see discussion in the lim- itations section) very easy. We hope that asking participants to be open about their methods can act as countermeasurg
s a is measure of inter-annotator agreement, for instance used in work on linguistic annotation (Paun et al., 2022). We use an evaluation measure by treating the system to be evaluated as one annotator, and the gold-standard creators as another. A perfect system always agrees with the gold-standard creators (a = 1), whereas a poor system shows no systematic agreement with them (a x 0). Even lower scores (-1 1 S a < 0)
Krippendorff's
are signs of systematic mistakes.
this potential problem. The design of the leaderboard is also guided by the idea that there no single best model (Etha- yarajh and Jurafsky, 2020). The choice of a model for some application sometimes need only depend on the performance on certain subsets of our eval- uation suite and can even be restricted to only in- cluded models up to a certain size. We therefore allow users of the leaderboard filter and sort out all the tasks and models that are irrelevant for their specific needs: users can include and exclude tasks, filter out model families, sort by size, overall performance, and task-specific performance. Consider the following use case: developer wants to build proofreading application using transformer model, and needs their application to run as fast as possible on limited hardware. The most relevant task in Superlim for them is DaLAJ-GED, and the developer may wish to re- strict themselves to models with less than 150M parameters. In the leaderboard, they can now eas- ily select all hase models, exclude all word-level tasks to declutter the table, and sort according to performance on DaLAI-GED. That would bring on top the KBLab/ber t-base- www.eeedishhhh-s---aaeee model, that otherwise performs worse than average. The developer further checks the performance on the validation set matches the performance on
Krippendorff's a is parameterized by a distance metric (Krippendorff, 10 which allows to use it different types of task. For binary and ternary labelling tasks, we use nominal-o. In the binary case, nominal-a linearly maps to macro-averaged f-score. For scoring tasks we use interval-o. Since it is a measure of agreement, interval-a is stricter than correlation measures like Pearson's 's r, which do not require a system to get the sizes of predicted scores correct, only the shape of their distribution. For a benchmark, find this strictness desirable property. The selection tasks do not allow for direct application of a. However, we can calcu- late derived measure, a pseudo-a, by pretending, after the fact, that the selection tasks are binary decision tasks and by calculating a nominal-o for this recasted task. We present a more elaborate investigation of as an evaluation measure in a separate paper (Bouma, 2023).
We currently do not have b-bede measure for the summarization task SweDN, for which the benchmark uses ROUGE-1.
Swe Winogender is evaluated using nominal-o, but in addition also using gender parity, which mea- sures how consistently the model answers across datapoints that only differ in the choice of pronoun.

---

Model
Type
Size
Language
Model
Avg
KB/bert-base-swerr basettbbb---wddis--csseel
BERT
base
Al-Nordicw/bentt-swwc
0.013
KBLab/bert- Bellabbbert-base-yee sestt-bbbss--stteeeeeeessesse--e swedist-cased-new
BERT
base
KBLLab/nt-bertttsww---16
0.018
KBLab//eegttron hetprregtttoo--ee--u-----sseeeee
pert-buue-sweddis---see----dd
BERT
base
KBBbbrrtb-sw-cc
0.018
KBI KBI hatpmmaauttroo-e-----------dddd A.Noriis/bett-lagg-swwdiii
165k
BERT
large
0.034
BERT
large
bbbitabbnn--ert-baaee
0.056
kIm-rohertahhan
RoRERTa
base
multi
xlm-roherta-base
0.085
xlm roberta-large
RoBERTa
large
multi KBLab/bert beseett---w---eeee swww--eww
0.122
AI- Sweden/gpt-sww
GPT 126M-400 sv, no, da, is,
xlm-roberta-
0.324
Table 2: Language models used for the reference implementation
Table Average standard devia- tion of performance across hyper-
parameter configurations
the test set, before finally choosing small set of suitable models evaluate on their new applica- tion's s data. A screenshot the leaderboard taken from this scenario can be found in Figure in Ap- pendix B.
Static word vectors? were used as approach SuperSim relatedness and SuperSim similarity by scaling the word pairs' cosine similarities to 0-10, SweSAT was addressed by picking the choice with the highest cosine similarity. For Swedish Analogy we first applied vector algebra to calculate vector for the fourth (missing) item, and then used cosine similarity to pick the closest vocabulary item.
7 Reference implementation
To create baselines for Superlim, we built a number of reference implementations using the Swedish language models listed in Table 2, all hosted at Hugginglaace In addition to these models, we also provide results for set of non-neural supervised machine learning methods, including SVM, Deci- sion Trees, and Random Forests. Random choice and majority label give lower bounds.
The two word-level selection tasks were also solved using the gpt-sw3-... models (Fkgren et al., 2023). Swedish Analogy by inspecting log- probabilities of a Swedish translation of the text synonym is being sought for the search term. The term x serves as synonym for y, where x is a choice item and the question's target expression, and choosing the most probable item. SweSAT employed few-shot learning strategy to prompt the GPT model to complete the analogy.
We used the following hyperparameters when fine tuning the language models, along with the default arguments in the HuggingFace Trainer c warm-up ratio 0.06, weight decay 0. (0.0 for GPT models), number of training epochs 10, fp16. We use early stopping with patience 5 for the number of epochs. Learning rate and batch size were tuned for all tasks and models, from the following search space: learning rate [le-5,2e-5,3e 5,3e--,33e-55 50e-5,,e--5 batch size [16,32]. Be- cause of the size of the training set, we only con- sidered the lowest and highest learning rate for SweNLI. The hyperparameter search gives us performance score (Krippendorff's a) for each task, model and configuration of hyperparame- ters. The average standard deviation of these scores are summarized in Table 3, which shows tarrrobbtta-lagge to be by far most sensitive to these settings.
Results and discussion
The results for the word-level tasks are in Table 4. The difference between SuperSim relatedness and SuperSim similarity for the static word vectors mir- rors the difference in the random bascline for these tasks, but it contrasts with results when measured in terms of correlation (Hengchen and Tahmasebi, 2021), which are much more even. Here the sensitivity of to the size of the predicted scores: cosine similarity systematically overesti- mates semantic similarity SuperSim similarity. The GPT models show a clear effect of model size on Swedish Analogy and SweSAT, and the largest models do well on these tasks. The smallest GPT model is however not more effective than static word vectors on SweSAT.
We created reference implementations for all text-level tasks except for SweDN.
Table 5 gives the results for the text-level tasks. The different tasks by and large "agree" with each
'huggingface, wtt.ccee.ooocc//eeeeeeeeee co/dbass/transffomerrs/mmiii classes/traie trainer

---

Word-level task
Model
SuperSim rel
SuperSim sim
Swedish Analogy
SweSAT
3wwcden/gpttsw33112000
0,400
0.340
Al J--Sweden/gpt-wwa -33
0.732
0,665
LL-Sweden/gpttsww33
0.824
0.820
AI- Al-Sweden/gpt-see -w-4
0.838
0.854
Random
-0.037 0.410
-0,302 -0.109
0.000
0.000
Static word vectors
0.013
0.365
Task type
score
score
select
select
Table 4: Reference model performance for the word-level tasks, reported as Krippendorff's a.
Text-level task
Model
Absabank Arg Sent
DaLAJ Paruphrase
FAQ
NLI
Winograd
Avg
Diagnostics
KBLLab/m--ertt-sss-ce.
0.508
0.628
0.753
0.874
0.777
0.231
0.308
0.189
0.534
0.393
Al-Nordics/bert-lswee
0,480
0.563
0.745
0.862
0.719
0.241
0.316
0.192
0.515
0.347
KB/bert-b-sw-c
0.529
0.555
0.740
0.845
0.641
0.179
0.376
0.139
0.501
0.349
klm-roherta-large
0.516
0.584
0.738
0.882
0.584
0.205
0.367
0.081
0.494
0,415
0.449
0.562
0.718
0.867
0.709
0.218
0.277
0.061
0.483
0.363
0.390
0.541
0.644
0.823
0.660
0.172
0.326
0.120
0.459
0.314
KBLab/bert-b b-s-ett-------ewww
0.428
0.554
0.753
0.755
0.447
0.163
0.140
0.042
0.410
0.338
Xlm-roberta-bass
0.366
0,497
0.701
0.813
0.473
0.186
-0.177
0,380
0,318
SVM
0.286
0.354
0.518
0.239
0.038
0.000
0.042
0.055
0.192
0.026
Decision Tree
0..17
0.156
0.269
0.200
0.040
0.192
0.040
+0.240
0.074
0.037
Random
0.008
0.013
0.007
-0,033
-0.150
-0.091
-0.010
0,081
-0.038
0.004
Random Forest
0,005
-0.272
-0.312
0.143
0.032
-0.411
0.003
-0.177
-0.124
0.010
Majority label/Avg
-0.052
-0.272
-0.340
-0.001
-0.310
-0.444
-0.333
-0.177
-0,240
-0.404
Task type
score
3-1ab
2-1ab
score
select
3-lab
2-lab
2-lab
3-lab
Table 5: Reference model performance for the text-level tasks, sorted average Krippendorff's a. and overall diagnostic score. See the main text for discussion of the SweWinogender results, here omitted for space reasons.
other in their ranking of the different models: mod- els scoring high on one task are likely to score high on another. We also see that the models character- ized as large on average outrank the other models. The scores are relatively low overall. We take this not just as an artefact of our choice of evaluation measure, which in most cases is stricter than, say, indication that
gender parity scores but very low of around -0.3. This is the result of outputting the same label for basically all test items. As mentioned above, SweNLI and the diagnostic tasks are inference tasks that share the same, automatically translated training dataset. It is tempting to assume perfor- mance is affected by low quality of the training set. We note, however, that Argumentation sentences also is ternary labelling task with automatically translated training data, and that the models fare much better there. likely that these are just hard tasks: also in Bowman and Dahl (2021) it is noted that GLUE's diagnostic set was challenging even for the best models, and SuperGLUE's co- reference task has the lowest baseline performance amongst the non-diagnostic tasks.
accuracy f-score, but also Superlim is difficult benchmark. We agree with Bowman and Dahl (2021) that this desirable property.
Particularly low scores are achieved for the in- ference task SweNLI and the pronoun resolution task SweWinogras The bad performance for the former is connected the low scores on both di- agnostic tasks, which use the same training data and are of the same task type. Overall performance on SweDiagnnsticc maxes out at o=.415, a break- down of SweDiagnostin results can be found in Appendix C.
The models perform best on the paraphrase task SweParaphrase.
Distribution
There is no column for the SweWinggender di- agnostic task in Table all models receive perfect
All datasets constituting Superlim are available under Creative Commons licenses (CC BY 4.0,

---

CC BY-SA 4.0, respectively), and can be down- loaded together with their documentation sheets from Språkbanken Text's website* or accessed through HuggingFace." The Superlim leaderboard can be found at KBLab's website," and includes model predictions well as detailed information about the participating models.
Acknowledgments
The Superlim project was supported by Sweden's s Innovation Agency (Vinnova, grant nos 2020- 02523 and 2021-04163 and by Nationella spräk- banken- - jointly funded by the Swedish Research Council (2018 8 8-2024, grant 2017-00626) and 10 partner institutions.
10 Conclusion
Joey Öhman contributed to Superlim while Al Sweden. The contributions of Simon Hengchen and Nina Tahmasebi have in part been funded by Towards Computational Lexical Semantic Change Detection (Swedish Research Council, 2019-20222 grant no. 2018-01 1844 and Change is Key! (Riks- bankens Jubileumsfond, grant no. M21-0021). RISE Research Institutes of Sweden was involved in the project at its early stage.
We have presented Superlim, a Swedish benchmark and analysis platform for natural language under- standing in the style of GLUE and SuperGLUEE Superlim contains 13 regular tasks and 2 diagnos- tic tasks from a broad range of task types, includ- ing word-level tasks that are not commonly seen in GLUE-style benchmarks. All tasks come with gold- standard test data, with training and development data albeit not always of gold standard quality and with detailed, standardized documentation.
We would like to thank everybody who con- tributed to the creation datasets or in any other way helped us with the project, in particular Francisca Hoyer, Yousuf Ali Mohammed, Tosin Adewumi, Julius Monsen, Arne Jönsson, Peter Ljunglöf and Jacobo Rouces.
We have chosen to make Superlim's leaderboard flexible and transparent, which we hope may help promote the dissemination of knowledge about the capacities of natural understanding models for Swedish and knowledge about their construction. Finally, by having comparable tasks the de facto standard SuperGLUE,, Superlim not only con- tributes to the development of natural language understanding for Swedish, but also to research into transfer learning and multilingual models, and thus to much wider part of the field of natural language processing.
Limitations
Unlike some other benchmarks, Superlim has no hidden data, which may lead to data leaks. The models (pre)trained on the data crawled from the Internet may have the test data. While we require the submissions to be transparent and thoroughly documented as possible, it is not always possible to control what exactly the very large train- ing sets contain (and often it is not even known by those who trained the model). There a risk that this problem will exacerbate as recent evidence have shown that larger language models (equal to or larger than 6B parameters), such as the GPT-J model (Wang and Komatsuzaki, 2021) and Chat- GPT,7 are capable of memorizing data verbatim (Carlini al., 2023). 8 In order better understand how this affects Superlim, we encourage future work exploration of how much of the Superlim data has been memorized by large language models.
Author contributions
The authors divided into three groups (Berdicevskis to Öhman, Adesam to Volodina, and Börjeson Tahmasebi), depending on their con- tribution the project and the paper. Within each group, the contributions are considered to be of equal weight, and the author names are listed al- phabetically. The division of labour between the three partner institutions was as follows: dataset collection and/or construction Språkbanken Text; reference implementation AI Sweden; leader- board development KBLab.
We do not currently have a human bascline to put the performance of models on Superlim into perspective.
Some of the training sets in the Superlim were automatically translated and have not been manu- ally corrected. The translated test sets have been
htttp:://spraakkaanke gu. se/resurrser/uperlli 'ht superlim-
/https://openai.comm http:/pppnaa..ccchaaaaaeeeoe /https://hitz-aa tit.... Im- contamination//llo//
'ht //tp.///abbbaa http:///....bb..eeeeeeerrrrrrrre see.//eeaarrrorrrreesssssss

---

thoroughly corrected, but, of course, they still may suffer to some extent from translationese (Geller- stam, 1986) or its exacerbated variant post-editese (Toral, 2019).
Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez- Gazpio, and Lucia Specia. 2017. 2001 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-220) 2017), pages 1-14, Vancouver, Canada. Association for Computational Linguistics.
It can be argued that Superlim is too heteroge- neous, containing, on the one hand more traditional natural language understanding tasks such as NLI, sentiment analysis, semantic similarity at the sen- tence level as well as word-level tasks, which are less often used to evaluate Transformer models, on the other (and one generation task in addition).
Robin Cooper, Dick Crouch, Jan Van Eijck. Chris Fox. Johan Van Genabith, Jan Jaspars, Hans Kamp. David Milward, Manfred Pinkal, Massimo Poesio, et al. 1996. Using the framework. Technical Report LRE 62-051 D-16, The FraCaS Consortium.
Ariel Figgren Amaru Cuba Gyllensten, Felix Stol- lenwerk, Joey Öhman, Tim Isbister, Evangelia Gogoulou, Fredrik Carlsson, Alice Heiman, Judit Casademont, and Magnus Sahlgren. 2023. GPT- SW3: An autoregressive language model for the Nordic languages.
References
Yvonne Adesam, Gerlof Bouma, and Richard Johans- son. 2015. Defining the Eukalyptus forest- the Koala treebank of Swedish. In Proceedings of the 20th Nordic Conference of Computational Linguistics (NODALIDA 20/5). pages 1-9, Vilnius, Lithuania. Linköping University Electronic Press, Sweden.
Kawin Ethayarajh and Dan Jurafsky. 2020. Utility is in the eye of the user: A critique of NLP leaderboards. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 4846-4853, Online. Association for Computa- tional Linguistics.
Tosin Adewumi, Foteini Liwicki, and Marcus Liwicki. 2022. Exploring Swedish & English fastTexx em- beddings. In Proceedings of the 8th International Workshop on Artificial Intelligence and Cognition, pages 201-208.
Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2002. Placing search in context: the concept revisited. ACM Trans. Inf. Syst., 10
Lukasz Augustyniak, Kamil Tagowski, Albert Sawczyn, Denis Janiak, Roman Bartusiak, Adrian Szymezak, Arkadiusz Janz, Piotr Szymański, Marcin Watroba, Mikol aj Morzy, Tomasz Kajdanowicz. and Maciej Piasecki. 2022. This is the way: designing and com- piling LEPISZCZE, a comprehensive NLP bench- mark for Polish. In Advances in Neural Information Processing Systems, volume 35, pages 21 1805-21818 Curran Associates, Inc.
Timnit Gebru, Jamie Morgenstern, Briana Vec- chione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2021. Datasheets for datasets. Commun. ACM, 64(12):86-92
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khy- athi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondfej Dušck, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jham- tani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bod- hisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan Major. Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan. Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura perez-Beltrachini Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodrigue7, Sashank Santhanam, João Sedoc, Thibault Sellam. Samira Shaikh, Anastasia Shimo- rina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021. The GEM benchmark: Natural language generation. its evaluation and metrics. In Proceedings of the 1st Workshop Natural Language Generation. Evaluation, and Metries (GEM 2021), pages 96-120, Online. Association for Computational Linguistics.
Lars Borin, Markus Forsberg, and Lennart Lönngren. 2013. SALDO: a touch of yin to WordNet's yang. Language resources and evaluation, 47(4) 1211.
Gerlof Bouma. 2023. Krippendorfi's a as an evaluation measure. [Unpublished manuscript] Sprákbanken Text, Department Swedish, Multilingualism, Lan- guage Technology, University of Gothenburg.
Samuel R. Bowman and George Dahl. 2021. What will it take to fix benehmarking in natural language under- standing? Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 4843-4855, Online. Association for Computational Linguistics.
Nicholas Carlini, Daphne Ippolito. Matthew Jagielski. Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. Quantifying memorization across neural lan- guage models. In The Eleventh International Confer- ence on Learning Representations.

---

Martin Gellerstam. 1986. Translationese in Swedish novels translated from English. Translation studies in Scandinavia, 1:88-95.
Klaus Krippendorft. 2018. Content Analysis: An In- troduction to Its Methodology, 4th edition. SAGE Publications.
Saga Hansson, Konstantinos Mavromatakis, Yvonne Adesam, Gerlof Bouma, and Dana Dannélls. 2021. The Swedish Winogender dataset. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pages 445-459) Reykjavik, Iceland (Online). Linköping University Electronic Press, Sweden.
Robin Kurtz. 2022. The KBLab blog: Evaluating Swedish language models.
Hector J. Levesque, Ernest Davis, and Leora Morgen- stern. 2012. The Winegrad schema challenge. In Proceedings of the Thirteenth International Confer- ence on Principles of Knowledge Representation and Reasoning, KR 12, page 552-561. AAAI Press.
Simon Hengchen and Nina Tahmasebi. 2021. Super- Sim: a test set for word similarity and relatedness in Swedish. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDa- LiDa), pages 268-275, Reykjavik, Iceland (Online). Linköping University Electronic Press, Sweden.
Peter Ljunglöf and Magdalena Siverbo. 2012. A bilin- gual treebank for the FraCaS test suite. In SLTC- 2012, 4th Swedish Language Technology Conference, Proceedings of the Conference.
Martin Malmsten, Love Börjeson, and Chris Haffenden. 2020. Playing with words at the National Library of Sweden making a Swedish BERT.
Felix Hill, Roi Reichart, and Anna Korhonen. 2015. ex-99: Evaluating Semantic Models With (Genuine) Similarity Estimation. Computational Lin- guistics, 6164))665-695
Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations vector space. In International Con- ference on Learning Representations.
Tim Isbister. Fredrik Carlsson, and Magnus Sahlgren. 2021. Should we stop training more monolingual models, and simply use machine translation instead? In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa). pages 385- 390, Reykjavik, Iceland (Online). Linköping Univer- sity Electronic Press, Sweden.
Julius Monsen and Ame Jönsson. 2021. A method for building non-English corpora for abstractive text summarization. In Proceedings of CLARIN Annual Conference 2021, pages 82-89.
Dan Nielsen. 2023. ScandEval: benchmark for Scan- dinavian natural language processing. In Proceed- ings of the 24th Nordic Conference on Computational Linguistics (NoDaliDa), pages 185-201, Tórshavn, Faroe Islands. University of Tartu Library.
Daniel Khashabi, Arman Cohan, Siamak Shakeri, Pe- dram Hosseini, Pouya Pezeshkpour, Malihe Alikhani, Moin Aminnaseri, Marzieh Bitaah, Faeze Brahman, Sarik Ghazarian, Mozhdeh Gheini, Arman Kabiri, Rabeeh Karimi Mahabagdi, Omid Memarrast, Ah- madreza Mosallanezhad, Erfan Noury, Shahab Raji. Mohammad Sadegh Rasooli, Sepideh Sadeghi, Er- fan Sadeqi Azer, Niloofar Safi Samghabadi, Mahsa Shafaei, Saher Sheybani, Ali Tazarv, and Yadollah Yaghoobzadeh. 2021. ParsiNLU: suite of language understanding challenges for Persian. Transactions of the Association for Computational Linguisties, 9:1147-1162.
Silviu Paun, Ron Artstein, and Massimo Poesio. 2022. Statistical Methods for Annotation Analysis. Springer Cham.
Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. WiC: the word-in-context dataset for evalu- ating context-sensitic meaning representations. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1267-1273, Minneapolis, Minnesota. Association for Computa- tional Linguistics.
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vid- gen. Grusha Prasad, Amanpreet Singh, Pratik Ring- shia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, 7eerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dynabench: Rethinking benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 41 10-4444, Online, Association for Computa- tional Linguistics.
Mahima Pushkarna, Andrew Zaldivar, and Oddur Kjar- tansson. 2022. Data cards: Purposeful and trans- parent dataset documentation for responsible AI. In 2022 ACM Conference Fairness, Accountability, and Transparency, FAceT '22. page 1776-1826. New York, NY, USA. Association for Computing Machin- ery.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Be- havioral testing of NLP models with CheckList In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4902- 4912. Online. Association for Computational Lin- guistics.
Klaus Krippendorff. 2011. dorff's alpha-reliability. https:/ httss://wwn www. asc. upenn. http://tte/deaall////iil/ Computingkkippendorfff sAlpha-Rell Reliability. pdf.
Computing Krippen-

---

Jacobo Rouces, Lars Borin, and Nina Tahmasebi. 2020. Creating an annotated corpus for aspect- based senti- ment analysis in Swedish. In DHN, pages 318-324.
Elena Volodina, lldikó Pilán, Stian Rødven Eide, and Hannes Heidarsson. 2014. You get what you an- notate: a pedagogically annotated corpus of course- books for Swedish a second language. In NEALT Proceedings Series, volume 22, pages 128-144.
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). pages 8-14, New Orleans, Louisiana. Association for Computational Linguistics.
Alex Wang. Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stick- ier benchmark for general-purpose language under- standing systems. Advances in neural information processing systems, 32.
Piotr Rybak, Robert Mroczkowski, Janusz Tracz, and
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018, GLUB: A multi-task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP,, pages 353-355, Brussels, Belgium. Association for Com- putational Linguistics.
Ireneusz Gawlik. 2020. KLEJ: Comprehensive benchmark for Polish language understanding. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages I 1191- 1201, Online. Association for Computational Linguis- tics.
David Samuel, Andrey Kutuzov, Samia Touileb, Erik Velldal, Lilja Øvrelid, Egil Ronningstad, Elina Sigdel, and Anna Palatkina. 2023. NorBench - a henchmark for Norwegian language models. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 618-633, Tórshavn, Faroe Islands. University of Tartu Library.
Ben Wang and Aran Komatsuzaki. 2021. GPT-J- 6B: A Billion Parameter Autoregressive Lan- guage Model. https://githh: cotbbbino///ooll////// mesh-trransormerr
Bryan Wilie. Karissa Vincentio, Genta Indra Winata, Samuel Cahyawijaya, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, and Ayu Purwarianti. 2020. IndoNLU: Benchmark and resources for evaluating Indonesian natural language understanding. In Proceedings of the 1st Conference of the -saa-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Lan- guage Processing, pages 843-857, Suzhou, China. Association for Computational Linguisties.
Tatiana Shavrina. Alena Fenogenova, Emelyanov An- ton, Denis Shevelev, Ekaterina Artemova, Valentin Malykh, Vladislav Mikhailov, Maria Tikhonova, An- drey Chertok, and Andrey Evlampiev. 2020. Rus- sianSuperGLUE: A Russian language understand- ing evaluation benchmark. In Proceedingg of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 4717-4720 Online. Association for Computational Linguistics.
Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS- MT - building open translation services for the world. In Proceedings of the 22nd Anmual Conference of the European Association for Machine Translation, pages 479-480, Lisboa, Portugal. European Associa- tion for Machine Translation.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A -overrae challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies, Volume (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Antonio Toral. 2019. Post-editese: an exacerbated translationese. In Proceedings of Machine Trans- lation Summit XVII: Research Track, pages 273-281, Dublin, Ireland. European Association for Machine Translation.
Beilei Xiang, Changbing Yang, Yu Li, Alex Warstadt, and Katharina Kann. 2021. CLiMP: A benchmark for Chinese language model evaluation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2784-2790, Online. Association for Computational Linguistics.
Dietrich Trautmann, Johannes Daxenberger, Christian Stab, Hinrich Schütze, and Iryna Gurevych. 2020. Fine-grained argument unit recognition and classi- fication. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05);90489900
Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao. Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang. Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang. Zhengliang Yang, Kyle Richardson, and Zhenzhong Lan. 2020.
Elena Volodina, Lena Granstedt, Arild Matsson, Beáta Megyesi, lldikó Pilán, Julia Prentice, Dan Rosén, Lisa Rudebeck, Carl- Schenström, Gunlög Sundberg, and Mats Wirén, 2019. The Swel.. lan- guage learner corpus: From design to annotation. Northern European Journal of Language Technology, 6:67-104.

---

Chinese language understanding evalua- tion benchmark. In Proceedings of the 28th Inter- national Conference on Computational Linguistics, pages 4762-4772, Barcelona, Spain (Online). Inter- national Committee on Computational Linguistics.
CLUE:

---

A Example Superlim-style documentation sheet
I. IDENTIFYING INFORMATION
Title*
Swe SweWinogender
Subtitle
A Swedish diagnostic set for gender bias in natural language inference models
Created by*
Yvonne Adesam a se),
Gerlof Bouma (gerlof. bouma@gu. se)
Publisher(s)*
Spräkbanken Text https://spraakbanken.. wttp:/nkennnn....eeeeeeeeeeeeeee
Link(s) / permanent
identifier(s)*
License(s)*
CC BY 4.0
Abstract*
The SweWinogender test set is diagnostic dataset to measure gender bias in coreference resolution/textual entailment. It is modelled after the English Winogender benchmark, and is released with reference statisties on the distribution of men and women between occupations and the association between gender and occupation in modern corpus material.
Funded by*
Vinnova (dnr 2020-02523, dor 2021-04165) Spräkbanken Text
Cite as Related datasets
[1]
Part of the Superlim collection (https://spraakbanken) gu. se/en/
resources/superling
See SweWinogender v1.0 for formulation of this task as pronoun
resolution problem.
Based upon/partially translated from Winogender Schemas [2]
USAGE Key applications
Diagnose gender bias in natural language inference systems (I) Indirectly the pronoun interpretation task cast as a natural language inference problem: decide whether a discourse fragment containing a pronoun entails a sentence with the pronoun replaced by a candidate antecedent.
Intended (aua(s)\)s)g((s)
(2) Compare system predictions between pronoun types (mascffem/gender neutral)
(3) Compare system predictions with auxiliary statisties on gender and occupation
Recommended evaluation
(1) Krippendorff's alpha on binary label
measures
2) "Gender parity": The proportion of triples of items differing only by
the type of pronoun, that receives identical labels. See [3].
(3) Correlation (Spearman's 's rho); plotting/visure inspection. See [2].
Dataset function(s) Recommended split(s)
Diagnosties
Test data only
III. DATA
Primary d
Text
Language*
Swedish
Dataset in n
624 test items from 104 templates. 312 positive cases ('entailment') and 312 negative cases ("neutral").

---

Nature of the content
'hee test items are constructed from short discourse templates that contain two participants: one referred by occupation, and one either by a role description. Furthermore, the templates contain a pronominal reference to one these participants. The templates are constructed such that the in- terpretation of the pronoun follows from (common sense) reasoning. Fach template gives risc to 6 test items: 3 possibilities depending on whether the feminine ("hon/henne/hennees), masculine ("han/honom/hans") or gender-neutral pronoun ("hen/hens")) is used, 2 possibilities depending whether the hypothesis is entailed not. A natural language inference model that is not sensitive to gender biases should therefore answer the same way for triple of test items that only differs in which pronoun they contain.
The test set accompanied by an auxiliary dataset that contains two sets of statistics on the association between occupation and gender for the occupations mentioned in the test set. These statistics were extracted from real-world database and from corpus, respectively. The auxiliary data can be used study gender-occupation biases in the system more directly. Test items: JSON Lines, with test item per line. Test items are given a pair of sentences prrmmise' and 'hypothesis') and 'label' attribute that says whether the hypothesis is entailed by the premise ( ('entailment') or not ("neutral'). The metadata ("meta') ) contains identifying information about the sentence template that generated the test item, and a 'tuple-id' that can be used to calculate parity.
#ormat
Auxiliary data: TSV file with one occupation per row. Gives the follow- ing columns of information: 1) occupation; 2) % female practitioners according to SCB; (3)-5) % occurences in female-associated contexts using small/mediun/lagge collocate sets. See [1] for an explanation of the different corpus measures.
Data source(s)* *
The test items are loose translations and/or inspired by the Winogender Schemes [2].
The auxiliary data was collected by the first authors of [1J, in the context of an MA course. The real-world statistics on gender and occupation were compiled on the basis of Statistics Sweden/SCB's open data (CC BY 4.0), Where occupations do not map 1-1 SCB's 's categorization scheme, the supplied statistics are averages over several relevant categories. See [1] for details. The corpus-based statisties on aassociiasoiiiiionn of occupations where compiled from the Swedish Culturomics Gigaword Corpus [4].
Data collection ms)
See [1]
Data selection and filtering*
See [1]
Data preprocessing**
See [1]
Data labeling*
Test items contain gold-standard coreference data by design. Test item compilation: native speaker of Swedish with PhD in computa- tional linguistics, 1 near-native speaker of Swedish with PhD in (corpus) inguistics.
Annotator characteristics
IV. ETHICS AND CAVEATS

---

Ethical considerations
The auxiliary data contains information about the distribution between women and men across occupations, and therefore contains data about subpopulations. The data does not contain reference to individuals neither directly nor indirectly.
Things to watch out for
This is meant as diagnostic, not as a target for training. The diagnostic only concerns occupation and gender, and this is only one of the many ways gender bias may be present in a coreference resolution model. In the words of [2]: "[a]s diagnostic test of gender bias, we view the schemas having high positive predictive value and low negative predictive value; that is, they may demonstrate the presence of gender bias system, but not prove its absence."
Although the test items contain a threeway distinction in the pronoums used (han [masc]. hon [fem], hen [neutral], the auxiliary data is restricted to a binary gender perspective. For task (3) above, it may however be interesting to compare system predictions for the gender-neutral pronoun ("hen") items with the auxiliary statistics to better understand how system handles resolution of this pronoun.
V. ABOUT DOCUMENTATION
Data last updated* Which changes have been made, compared to the previous v
20230125 v2.0
Reformulation as a natural language inference task.
Access to previous versions
Earlier versions available from website.
This document created*
20210614; Gerlof Bouma (gerlof.bouma@guu se) 20230208; Gerlof Bouma (gerlof bouma@gu. se)
This document last updated*
Where to look for further details
Documentation template
v1.1
v
VI. OTHER
Related projects
References
[1] Hansson, Mavromatakis, Adesam, Bouma and Dannélls (2021): The
Swedish Winogender Dataset. In Proceedings of the 23rd Nordic Con- ference on Computational Linguistics (NoDaLiDa), pp452-459, http: / //www.ep www.epp.1r... hlluss//eccp/////8//////222222 1aap:/222211112..ppp .dd
[2] Rudinger, Naradowsky, Leonard and Van Durme (2018): Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp8-14. https://doi.org: 1163.0..111133111111 2002
[3] Wang, Pruksachatkun, Nangia, Singh, Michael, Hill, Levy and Bow- man (2019): SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. Advances in Neural Information Processing Systems 32. https: //tp:::///pppprr/ nips. cc/ pttppppppr/2//////// 1//ffl/////////// 4496bf 4.96ff224fe/7aaafa444444222266 Papeee..pdd-

---

[4] Rødven Eide. Tahmasebi and Borin (2016): The Swedish Culturomics Gigaword corpus: A one billion word Swedish reference dataset for NLP. In Digital Humanities 2016. From Digitization to Knowledge: Resources and Methods for Semantic Processing of Digital Works/Texts, pp8-12. Sttps://ep liu. stt. ec/ecp/ 126 /2tp///cccppppce/// 212602..pdf
B Leaderboard
SUPERLIM LEADERBORRD
ABOUT REBULTS TASKS DOG9 BUENT
ABSAbink Argumentziion
Relatedness
Binilaityy
-wedish
Dataset Swe
Sel: SuwParaphrase SywGnT-Syrenyeee
Swelliinggettee SwwtWinggrd
Poona
A264
SwetA0
Kitbel undia 2014
Froade
99M
3098
64008
3013
3.00
3716
0156
0751
6an
Freede
2088
20000
1308
1927
6773
6m
Showing to entree (ffineed from total nett
Kangkge
Figure I: A screenshot of the website showing the leaderboard, filtered to show only large language models of the 'base' category, sorted by performance the DaLAJ-GBE task on the validation (dev) set, with datasets for which the results are not available excluded.
C SweDiagnostics: detailed results
Coarse-grained categories
Fine-grained categories
Model
All
LS
PAS
L
K
UQnt
MNeg
2Neg
Coref
Restr
Down
Alm-roberta-large
0.415
0.441
0.434
0.345
0.290
0.509
0.648
0.641
0.391
0.167
-0.541
KBLab/mt- heattmttt-eerr---eeeee bewt-l-sw-e- 165k
0.393
0.368
0.430
0.305
0.314
0.669
0.434
0.641
0.334
0.115
-0.541
0.363
0.303
0.392
0.306
0.283
0.669
0.43
0.641
0.365
0.000
-0.621
KB/bert-b-sw-a
0.349
0.319
0.354
0.282
0.273
0.582
0.366
0.433
0.349
-0.114
-0.348
Al-Nordics/bettt-swcc
0.347
0.310
0.348
0.308
0.239
0.510
0.646
0.799
0.391
-0.114
-0.548
Bel.aWhhrt-b-sw-c-ne
0.338
0.327
0.387
0.277
0.214
0.513
0.505
0.571
0.361
-0.114
-0.528
xlm-roberta-bon base
0.318
0.307
0.359
0.245
0.180
0.500
0.145
0.107
0.310
-0.116
-0.561
0.314
0.305
0.358
0.270
0.144
0.582
0.578
0.339
0.262
-0.057
-0.463
Decision tree SVM
0.037
0.000
0.041
0.047
0.028
0.555
-0.247
-0.141
0.113
0.266
-0.169
0.026
-0.006
0.000
0.047
0.003
0.319
-0.366
-0.429
-0.032
0.080
-0,082
Random forest
0.010
-0.034
-0.052
0.006
0.067
0.347
-0.284
0.328
-0.099
-0.127
-0.169
Random
0.004
-0.005
0.040
0.013
0.020
0.341
0.004
-0.086
0.048
-0.089
-0.119
Majority label/Avg
-0.404
-0.378
-0.482
00.76
-0.350
-0.00
00.51
-0.626
-0.411
-0.600
-0.579
Table 7: The results the SweDiagnosties dataset. We report Krippendorf's the coarse-grained categories Lexical Semantics (LSS), Predicate-Argumer Structure (PAS), Logic (L) as well as Knowledge (K). In addition, five fine- graiggr categories, Universal Quantifiers (UQnt), Morphological Negation (MNeg), Double Negation (2Neg), Anaphora/Corefen Coreference (Coref), Restrictivity (Restr) and Downward Monotone (Down) are selected. The table reports categories identical to the ones reported in Table in the GLUE paper (Wang et al., 2018 8), but note the difference in measure, which was Ra in the cited paper.


Reviews:
[{"review_id": "sM8Sw6mxvW", "reviewer": "Reviewer_4JdJ", "summary": "", "questions": "Question A) Regarding line 238, could you provide more detail on how the data was \"culturally adapted\", in terms of who did it and what process they followed? I think that some of these details should be included in the text.\n\nQuestion B) Regarding line 245, why was OPUS-MT used? I'm not directly familiar with the system but I suspect that its performance is subpar. This is especially confusing to me because the Google Translate API was used on another dataset.\n\nQuestion C) For all datasets where something was translated or post-edited by one or more native speakers, can you provide more details on who those native speakers were? Were they authors of this paper, colleagues, professional translators, bilingual crowd-workers, or someone else? The resulting quality is likely to depend on this. This is especially important for the discussion at line 344.\n\nQuestion D) What exactly are the values in Table 3? I gathered that the average is taken over various hyperparameter settings for a given model, but what distributions are you calculating standard deviations for? Are they, for example, distributions of mean benchmark scores for some number of randomized training runs?\n\nQuestion E) Have you taken steps to mitigate the risk of accidental data leakage? I accept that preventing intentional cheating is out-of-scope, but from a practical standpoint it would be nice to know that accidentally training on test data is difficult. For example, putting the test inputs and their outputs in separate files might mitigate a web-crawl corpus from meaningfully leaking their correspondence.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments. Below we will answer the direct questions posed by the reviewer. When it comes to the questions regarding specific datasets, we want to emphasize that this information will also be documented in the datasheets, which are part of the AnonymizedBenchmark project and will be linked to in the paper.\n\n__Question A, cultural adaptation of anon-texttask-7__ The data was culturally adapted by a native of speaker of Swedish, professor in NLP, by replacing English proper names (e.g. person names, toponyms, trademarks) and some other terms (e.g. job titles) with idiomatic Swedish equivalents.\n\t \n__Question B, choice for OPUS-MT__ \nThe processing of different datasets was carried out by different researchers at different points in time, and while there were universal project policies, some decisions were made independently. This is why it was possible for different machine translation systems to have been used. In this particular case, OPUS-MT was chosen because it is an open, transparent and accessible MT system, that gives reproducible results, which was taken as a more important characteristic than the possible improvement of translation quality that Google Translate potentially would offer. \n\t \n__Question C, translators/post-editors__ (For reasons of anonymity, we will not comment on whether dataset creators were co-authors, colleagues, etc.)\n* Anon-texttask-2 was post-edited by a native speaker of Swedish with a Master in language technology.\n* Anon-texttask-4 was post-edited by a native speaker of Swedish, a graduate student with a background in linguistics\n* Anon-texttask-7 was corrected and culturally adapted by a professor of language technology, who is a native speaker of Swedish with a PhD in linguistics.\n* Anon-texttask-9 was translated from the English / based upon English examples in committee by a native speaker and a near-native speaker  of Swedish with PhDs in (computational) linguistics.\n* Anon-diagnose-1 was post-edited by a native speaker of Swedish with a Master in linguistics. \t\n* Anon-diagnose-2 was translated from the English / based upon English examples in committee by a native speaker and a near-native speaker of Swedish with PhDs in (computational) linguistics.\n\n__Question D, Table 3__ For a given type of LLM and a given text-level task, there are 8 runs (2 batch sizes x 4 learning rates), each run gives a performance score as Krippendorff alpha. We then calculated the std dev over these runs. This tells us how much the performance of a single model on a single task varies with different hyperparameter settings, that is, it is an indication of how sensitive the model is to its hyperparameter settings on that task. The reported numbers in Table 3 give a summarized hyperparameter sensitivity score for each model by averaging over these task-specific std devs. We agree this is presented rather densely in the paper and will make an effort to improve the discussion of these results for the final paper.\n\n__Question E, data leakage__ Since many of the datasets are projects which have been released independently, we cannot guarantee that they have not already been collected by web crawlers. We will, however, take the reviewer's suggestion and split the labels into separate files on our end. Furthermore, we will encourage investigations into memorization of input and output data on the models using our benchmarks, in order to estimate the effect of accidental training on testing data. The following citation would be a good inspiration for this kind of work: Carlini, Nicholas, et al. \"Quantifying memorization across neural language models.\" arXiv preprint arXiv:2202.07646 (2022). We will mention this in the final version of the paper.\n\nFinally, we share the reviewers concern that there can be a negative impact of using post-edited machine translated data, even though we do not know the degree of this impact. We have tried to lessen this impact on the benchmark by also including original Swedish datasets, and by conscious efforts of our translators/post-editors to make the translations sound more natural. Given the limited available annotated data in Swedish and the resources available for this project, it would not have been possible to include the current number and range of tasks without using post-edited data at all. The mentioned citation of Toral (2019) would make an excellent addition to the discussion in the Limitations section of our paper.\n\nIn a final version, we will address the grammar and presentation issues that the reviewer raised.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": " - NLP benchmarks in non-English languages are sorely needed.\n - The proposed benchmark's tasks provide broad coverage within the NLU domain.\n - The features included in the leaderboard and discussion behind the dataset documentation paradigm indicate that the authors thoughtfully considered the needs of multiple groups, including system developers and users.\n - The tasks and their source datasets are described with sufficient detail.", "reasons_to_reject": " - The potential negative impact of using post-edited test data may be underestimated. See the Missing References section.", "questions_to_authors": "Question A) Regarding line 238, could you provide more detail on how the data was \"culturally adapted\", in terms of who did it and what process they followed? I think that some of these details should be included in the text.\n\nQuestion B) Regarding line 245, why was OPUS-MT used? I'm not directly familiar with the system but I suspect that its performance is subpar. This is especially confusing to me because the Google Translate API was used on another dataset.\n\nQuestion C) For all datasets where something was translated or post-edited by one or more native speakers, can you provide more details on who those native speakers were? Were they authors of this paper, colleagues, professional translators, bilingual crowd-workers, or someone else? The resulting quality is likely to depend on this. This is especially important for the discussion at line 344.\n\nQuestion D) What exactly are the values in Table 3? I gathered that the average is taken over various hyperparameter settings for a given model, but what distributions are you calculating standard deviations for? Are they, for example, distributions of mean benchmark scores for some number of randomized training runs?\n\nQuestion E) Have you taken steps to mitigate the risk of accidental data leakage? I accept that preventing intentional cheating is out-of-scope, but from a practical standpoint it would be nice to know that accidentally training on test data is difficult. For example, putting the test inputs and their outputs in separate files might mitigate a web-crawl corpus from meaningfully leaking their correspondence.", "paper_topic_and_main_contributions": "This paper presents a large multi-task benchmark for language understanding in Swedish, loosely similar to SuperGLUE, publicly released by the authors. The paper describes  the tasks and origin of the data, and the authors have developed a feature-rich leaderboard for tracking progress on the benchmark. The authors propose an evaluation measure that scores all tasks (with the exception of one) on the same scale. They also propose and utilize a new paradigm for documenting datasets, intended to incur lower overhead than existing paradigms. Finally, they provide reference implementations for most tasks and present baseline results.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Many of your datasets include machine-translated text that was post-edited by a human, and the introduction claims that they are gold-standard quality just like from-scratch annotations. While this is briefly addressed in the Limitations section (which I appreciate), I think you might be under-estimating the difference between post-edited MT and natural text. A citation to consider is:\n\nAntonio Toral. 2019. Post-editese: an Exacerbated Translationese. In Proceedings of Machine Translation Summit XVII: Research Track, pages 273–281, Dublin, Ireland. European Association for Machine Translation.", "Typos_Grammar_Style_and_Presentation_Improvements": " - Line 13: I'm not sure what a \"smaller language\" is. I think readability would improve with a more precise phrase, such as \"lower-resource language\" or \"less-supported language\" (assuming I inferred your intent correctly).\n - Table 1: When I first looked at the bottom row, I didn't understand that I was looking at ditto marks (''). I later figured it out, and you don't necessarily need to change it, but I would recommend to double-check that your motivation is to improve clarity/readability instead of e.g. saving keystrokes or ink.\n - Line 35: \"Anonymized Namea\" -> \"AnonymizedName, a\"\n - Line 119: \"In stead\" -> \"Instead\"\n - Line 195: \"extend\" -> \"extent\"\n - Line 196: I think the idiom \"state of affairs\" hurts readability here. Consider just \"meaning\" or some other more direct phrase.\n - Table 3: The caption is not clear and is probably missing one or more words between \"deviation\" and \"hyperparameter\".\n - Lines 616-617: \"information the\" -> \"information on the\"\n - Line 645: \"lead data\" -> \"lead to data\""}}, {"review_id": "LXbZw3skg3", "reviewer": "Reviewer_kEqu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments. The reviewer raises concerns about the lack of information in the paper about dataset creation and reliability. \n\nIndeed, as the reviewer speculates, the descriptions of the tasks and their datasets are kept short because of the length of the paper. It is also related to the style of the paper, which is intended as a high level overview of the whole benchmark consisting of 13+2 tasks representing a range of NLU problems, and not as a detailed description of each task. We have therefore restricted details about the datasets to the documentation sheets. In addition, we note that some of the tasks already have their own associated publications with more detailed descriptions. This especially holds for included datasets that were created outside of the project.  \n\nHaving said that, we agree strongly with the reviewer that having detailed information about the annotation/construction process, and about the quality and reliability of the datasets is very important. The standardized documentation sheets specified for the benchmark were partly motivated by the desire to ensure such information is available and easy to locate.\n\nWe will here answer the questions about the specific datasets raised by the reviewer, but reiterate that this information will be found in the documentation sheets. We will make sure to be clearer about this in the paper.\n\n* __anon-texttask-8, Eukalyptus corpus__ The Eukalyptus treebank as used for anon-texttask-8 has sense annotation for about ¾ of its ~100k tokens, each sense label accompanied by a quality indication. For anon-texttask-8, we only used the tokens labeled with the highest quality levels, where a) all of two or more annotators were unanimous, or b) the label had been vetted/corrected after a first annotation round by the leader of the annotation project.\n* __anon-wordtask-1 and anon-wordtask-2__ The data was taken as-is from the SuperSim dataset (Hengchen and Tahmasebi, 2021). This paper reports average pairwise Spearman’s ρ = 0.67 for similarity and ρ = 0.73 for relatedness. \n* __anon-wordtask-4__  The Swedish Scholastic Aptitude Test (Högskoleprovet) is a national test, held twice each year. We have taken one part of the Swedish SAT, the multiple choice synonym test, as the basis for our anon-word-task-4, from 15 years of past tests. These items, including the _by construction_ correct answers, are available from a website run by the government agency responsible for the SAT (the Swedish Council for Higher Education). No further selection/annotation was performed in the compilation of anon-wordtask-4. \n\nRegarding IAA information in the documentation sheets: there is no designated heading called \"inter annotation agreement\" or similar, but there _is_ room for this and related types of information in the documentation sheet, under the row headings \"Data collection methods\", \"Data labeling\", and \"Annotator characteristics\" (final rows in section III of the sheet), depending on the precise nature and context of the information.\n\nThere is no mention of IAA in the example sheet in Appendix A, since IAA is not relevant for this dataset. It was not created by labelling/annotating given items, but rather by purposefully constructing and manipulating test items to have a preferred reading that either follows or is at odds with a gender-related bias. However other documentation sheets will include this information where relevant.\n\nIn the final version of the paper, we will address the grammar and presentation issues raised by the reviewer. In particular we will fix the formulation of the the opening sentence to better communicate the intended meaning, which relates to the rise of LLMs that can be applied to a wide range of tasks and therefore call for benchmarks that deal with such a wide range.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The range of selected tasks is sufficiently wide, it follows other benchmarking suites (e.g. GLUE) and overall seems to be well maintained -- I definitely support the \"no hidden data\" approach (NLP simply needs more evaluation data -- even if it is costly, not hiding parts of any existing ones as a lame excuse) as well as policy for the leaderboard as explained in the paper.", "reasons_to_reject": "My only strong objections are related to the gold data creation and its reliability in terms of inter-annotator agreement. In the end the gold data is what everything depends on and it seem not to be described in very detail in the paper -- though I understand this is partially because of length limits. This concerns mainly the word-level tasks:\n- anon-texttask-8, where sentences are taken from the Eukalyptus corpus (the 2016 paper on Eukalyptus corpus mentions only a fraction [\"As of October, 2015, two annotators have annotated a total amount of 17,580 tokens, with an overlap of 2,919 tokens.\"] actually having more than one annotation. Has that changed/improved meanwhile? Even two annotations are a very poor standard, and I believe that by 2023 no single-person-annotated data should be part of any evaluation sets, particularly when it concerns tasks with a notoriously low inter-annotator agreement like WSD.\n- anon-wordtask-1 and anon-wordtask-2, where you say \"labelled by five native speakers of Swedish\". What was their agreement/correlation?\n- anon-wordtask-4 is missing any details how the SweSAT was used (how words were chosen, who annotated, what was the IAA -- if at all available)\n\nSimilarly, information about inter-annotator agreement (i.e. how many raters + raw numbers + maybe some chance-corrected coefficient) is completely missing from the documentation sheet (while I think it is the second most important quantitative information right after size :). Ideally all the tasks should be accompanied with information about human evaluation (where available).\n\nOnce the benchmark is here, many people will use it, and not many will look into it (unfortunately). I find it therefore important to raise these concerns now.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents na new set of NLP tasks proposed as benchmark for evaluating general-purpose language models.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- first paragraph in Introduction: \"In recent years, the move towards Transformer based large pretrained language models has created a need for comprehensive and standardized benchmarking in NLP\". Really? I do see you need to start the paper somehow, but starting it with a sentence that is basically false seems to be really awkward. The need has been there forever, not any smaller than these days!\n- l.119 extra space in \"In stead\"\n- l.418 the use OF? a...\n- l.617 ABOUT? the participating models"}}, {"review_id": "j6fqy18kSo", "reviewer": "Reviewer_kbFp", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive comments and will fix the raised grammar and presentation issues in a final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written, presents a sound work, and makes very interesting comments on the methodology used to build the benchmark and the choice of the datasets, the limitation of using a single measure to compare models, and the anglocentric bias. ", "reasons_to_reject": "I don't see any reason to reject the paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a benchmark for the Swedish language models, inspired by GLUE and SuperGLUE. The authors detail the features of the benchmark, the collection and annotation of datasets used, and exemplify the use of such benchmark. This is a significant progress towards better Swedish models evaluation and development. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- the unit of dataset sizes is not given in Table 1\n- in stead -> instead \n- ten different text -> ten different texts\n- thedataset -> the dataset \n- it performance -> its performance"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents a multi-task NLP benchmark and analysis platform for evaluating Swedish language models. 

The three reviewers agree on the strong/excellent quality of the work with respect to soundness and excitement. The reviewers have performed a thorough work and the authors provided clear responses during rebuttal. The only relevant concern has to do with some remaining doubts on inter-annotator agreement. This information should be clearly stated in the paper, together with all other minor comments raising during the discussion period.
Decision: Accept-Main