Repository: ScandEval/ScandEval (language: python)

Issue:
Allow different label schemes for each dataset
Currently a task has a set label scheme, and we enforce that each dataset with that task follows that label scheme. This involves things like mapping OntoNotes NER tags to CoNLL NER tags.

The point of this was originally to make the benchmarking on a task the same for each language, but as argued in [this discussion](https://github.com/ScandEval/ScandEval/discussions/85), this is less of a concern now. We would thus like to "free the labels" of all affected datasets, which involves (a) making it possible for the framework to handle different label schemes per dataset, and (b) to re-benchmark all the affected (model, dataset) combinations.


Discussion / hints:

Fix patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 67c67d966..e51d40cf9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -24,7 +24,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
   cached model outputs or datasets.
 - Added the following new datasets:
     - `fone`, a Faroese NER dataset, which replaces the previous `wikiann-fo` dataset.
-    - `dansk`, a Danish NER dataset, which complements the previous `dane` dataset.
+    - `dansk`, a Danish NER dataset, which replaces the previous `dane` dataset.
     - `norquad`, a Norwegian question answering dataset, which replaces the previous
       `scandiqa-no` dataset.
     - Danish, Swedish, German and Dutch versions of the MMLU, ARC and HellaSwag
diff --git a/src/scandeval/dataset_configs.py b/src/scandeval/dataset_configs.py
index c86f92bf9..bf9b78102 100644
--- a/src/scandeval/dataset_configs.py
+++ b/src/scandeval/dataset_configs.py
@@ -200,29 +200,6 @@ def get_dataset_config(dataset_name: str) -> DatasetConfig:
     max_generated_tokens=128,
 )
 
-DANE_CONFIG = DatasetConfig(
-    name="dane",
-    pretty_name="the truncated version of DaNE",
-    huggingface_id="ScandEval/dane-mini",
-    task=NER,
-    languages=[DA],
-    prompt_prefix="Følgende er sætninger og JSON-ordbøger med de navngivne enheder, "
-    "som forekommer i den givne sætning.",
-    prompt_template="Sætning: {text}\nNavngivne enheder: {label}",
-    prompt_label_mapping={
-        "b-per": "person",
-        "i-per": "person",
-        "b-loc": "sted",
-        "i-loc": "sted",
-        "b-org": "organisation",
-        "i-org": "organisation",
-        "b-misc": "diverse",
-        "i-misc": "diverse",
-    },
-    num_few_shot_examples=8,
-    max_generated_tokens=128,
-)
-
 DANSK_CONFIG = DatasetConfig(
     name="dansk",
     pretty_name="the truncated version of DANSK",


Test patch:
diff --git a/tests/test_named_entity_recognition.py b/tests/test_named_entity_recognition.py
index 26c7aec25..9b9e1f3fa 100644
--- a/tests/test_named_entity_recognition.py
+++ b/tests/test_named_entity_recognition.py
@@ -8,7 +8,6 @@
 from scandeval.dataset_configs import (
     CONLL_EN_CONFIG,
     CONLL_NL_CONFIG,
-    DANE_CONFIG,
     DANSK_CONFIG,
     FONE_CONFIG,
     GERMEVAL_CONFIG,
@@ -25,7 +24,6 @@
 @pytest.fixture(
     scope="module",
     params=[
-        DANE_CONFIG,
         DANSK_CONFIG,
         SUC3_CONFIG,
         NORNE_NB_CONFIG,
@@ -37,7 +35,6 @@
         CONLL_EN_CONFIG,
     ],
     ids=[
-        "dane",
         "dansk",
         "suc3",
         "norne_nb",