Repository: flairNLP/flair
Pull request #3639: Add Danish NER corpus
Author: login: alanakbik
Created: 2025-03-13T14:34:13Z | Merged: 2025-03-22T11:00:03Z

This PR adds the DANSK corpus for NER in Danish.


Diff:
diff --git a/flair/datasets/__init__.py b/flair/datasets/__init__.py
index c64891898b..f1c1279355 100644
--- a/flair/datasets/__init__.py
+++ b/flair/datasets/__init__.py
@@ -188,6 +188,7 @@
     NER_BAVARIAN_WIKI,
     NER_CHINESE_WEIBO,
     NER_DANISH_DANE,
+    NER_DANISH_DANSK,
     NER_ENGLISH_MOVIE_COMPLEX,
     NER_ENGLISH_MOVIE_SIMPLE,
     NER_ENGLISH_PERSON,
diff --git a/flair/datasets/sequence_labeling.py b/flair/datasets/sequence_labeling.py
index 81bce21a8a..578d7dcc80 100644
--- a/flair/datasets/sequence_labeling.py
+++ b/flair/datasets/sequence_labeling.py
@@ -5618,3 +5618,109 @@ def __init__(
             label_name_map=label_name_map,
             **corpusargs,
         )
+
+
+class NER_DANISH_DANSK(ColumnCorpus):
+    """Danish NER dataset from the chcaa/dansk-ner HuggingFace dataset."""
+
+    def __init__(
+        self,
+        base_path: Optional[Union[str, Path]] = None,
+        in_memory: bool = True,
+        **corpusargs,
+    ) -> None:
+        """Initialize the dansk-ner corpus.
+
+        Args:
+            base_path: Path to the corpus on your machine
+            in_memory: If True, keeps dataset in memory giving speedups in training
+            corpusargs: Additional arguments for corpus initialization
+        """
+        if base_path is None:
+            base_path = Path(flair.cache_root) / "datasets" / "ner_danish_dansk"
+        else:
+            base_path = Path(base_path)
+
+        # Create the corpus directory if it doesn't exist
+        base_path.mkdir(parents=True, exist_ok=True)
+
+        # Download dataset from HuggingFace and convert to CoNLL format
+        for split in ["train", "test", "validation"]:
+            conll_path = base_path / f"{split}.tsv"
+
+            # Only download and convert if file doesn't exist
+            if not conll_path.exists():
+                try:
+                    from datasets import load_dataset
+
+                    # Load the specific split from HuggingFace
+                    ds = load_dataset("chcaa/dansk-ner")[split if split != "validation" else "dev"]
+
+                    # Serialize the dataset to JSON for debugging
+                    debug_json_path = base_path / f"{split}_debug.json"
+                    import json
+
+                    # Convert dataset to a list of dictionaries and save with nice formatting
+                    dataset_for_json = [
+                        {"text": item["text"], "tokens": item["tokens"], "ents": item["ents"]} for item in ds
+                    ]
+
+                    with open(debug_json_path, "w", encoding="utf-8") as f_debug:
+                        json.dump(dataset_for_json, f_debug, ensure_ascii=False, indent=2)
+
+                    # Convert to CoNLL format
+                    with open(conll_path, "w", encoding="utf-8") as f_out:
+                        for example in ds:
+                            text = example["text"]  # Don't strip the text
+                            tokens = example["tokens"]
+                            ents = example["ents"]
+
+                            # Create token-level tags (default to 'O')
+                            tags = ["O"] * len(tokens)
+
+                            # Assign BIO tags based on entity positions
+                            for ent in ents:
+                                start_char = ent["start"]
+                                end_char = ent["end"]
+                                ent_label = ent["label"]
+
+                                # Find tokens that overlap with this entity
+                                for i, token in enumerate(tokens):
+                                    token_start = token["start"]
+                                    token_end = token["end"]
+
+                                    # If token overlaps with entity
+                                    if token_start >= start_char and token_end <= end_char:
+                                        # First token gets B- prefix
+                                        if token_start == start_char:
+                                            tags[i] = f"B-{ent_label}"
+                                        # Subsequent tokens get I- prefix
+                                        else:
+                                            tags[i] = f"I-{ent_label}"
+
+                            # Write tokens and tags
+                            for token, tag in zip(tokens, tags):
+                                token_text = text[token["start"] : token["end"]]  # Don't strip the token
+                                if token_text:  # Still skip empty tokens
+                                    # Replace newlines with space in output to maintain CoNLL format
+                                    token_text = token_text.replace("\n", " ")
+                                    f_out.write(f"{token_text}\t{tag}\n")
+
+                            # Empty line between sentences
+                            f_out.write("\n")
+
+                except Exception as e:
+                    print(f"Error downloading or converting dataset: {e}")
+                    raise
+
+        # Initialize corpus using the converted files
+        super().__init__(
+            base_path,
+            column_format={0: "text", 1: "ner"},
+            train_file="train.tsv",
+            test_file="test.tsv",
+            dev_file="validation.tsv",
+            column_delimiter="\t",
+            in_memory=in_memory,
+            **corpusargs,
+        )


Reviews:
user: login: stefan-it
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T08:54:01Z

Review comments:
diff_hunk: @@ -5618,3 +5618,109 @@ def __init__(
             label_name_map=label_name_map,
             **corpusargs,
         )
+
+
+class NER_DANISH_DANSK(ColumnCorpus):
+    """Danish NER dataset from the chcaa/dansk-ner HuggingFace dataset."""
+
+    def __init__(
+        self,
+        base_path: Optional[Union[str, Path]] = None,
+        in_memory: bool = True,
+        **corpusargs,
+    ) -> None:
+        """Initialize the dansk-ner corpus.
+
+        Args:
+            base_path: Path to the corpus on your machine
+            in_memory: If True, keeps dataset in memory giving speedups in training
+            corpusargs: Additional arguments for corpus initialization
+        """
+        if base_path is None:
+            base_path = Path(flair.cache_root) / "datasets" / "ner_danish_dansk"
+        else:
+            base_path = Path(base_path)
+
+        # Create the corpus directory if it doesn't exist
+        base_path.mkdir(parents=True, exist_ok=True)
+
+        # Download dataset from HuggingFace and convert to CoNLL format
+        for split in ["train", "test", "validation"]:
+            conll_path = base_path / f"{split}.tsv"
+
+            # Only download and convert if file doesn't exist
+            if not conll_path.exists():
+                try:
+                    from datasets import load_dataset
+
+                    # Load the specific split from HuggingFace
+                    ds = load_dataset("chcaa/dansk-ner")[split if split != "validation" else "dev"]
+
+                    # Serialize the dataset to JSON for debugging
+                    debug_json_path = base_path / f"{split}_debug.json"
+                    import json
+
+                    # Convert dataset to a list of dictionaries and save with nice formatting
+                    dataset_for_json = [
+                        {"text": item["text"], "tokens": item["tokens"], "ents": item["ents"]} for item in ds
+                    ]
+
+                    with open(debug_json_path, "w", encoding="utf-8") as f_debug:
+                        json.dump(dataset_for_json, f_debug, ensure_ascii=False, indent=2)
path: flair/datasets/sequence_labeling.py
user: login: stefan-it
body: I think lines 5659 - 5669 can be removed due to debugging-only :)
created_at: 2025-03-18T08:54:01Z
author_association: MEMBER

PR comments:
user: login: alanakbik
created_at: 2025-03-13T15:42:35Z
body: @stefan-it there are some discrepancies in the dataset statistics, but I think this is probably explained by the removal of the OpenSubtitles part of the corpus, as noted here: https://huggingface.co/datasets/chcaa/dansk-ner#domain-distribution
author_association: COLLABORATOR
user: login: stefan-it
created_at: 2025-03-17T09:38:51Z
body: Hey @alanakbik , thanks for this PR!

I compared it quickly to my draft implementation, and I spotted a few differences - right side is output of this PR, training split:

![image](https://github.com/user-attachments/assets/ff73eb7e-1b91-4a98-8a9e-920baa9278cd)

![image](https://github.com/user-attachments/assets/9e846050-f0db-46c1-ac5a-878100c20e02)

 or:

![image](https://github.com/user-attachments/assets/44c558ab-4239-43b7-ac5e-6a5be38bd447)
author_association: MEMBER
user: login: alanakbik
created_at: 2025-03-18T01:27:31Z
body: @stefan-it thanks for noticing this! There was an issue with handling of newline characters that were previously simply excluded, causing the token offsets to not line up anymore. The first sentence in your example was prefaced by a single newline token, which was excluded, explaining the offsets. 

With this fix, the statistics of the dataset now fully line up with the statistics posted on https://huggingface.co/datasets/chcaa/dansk-ner#domain-distribution, minus the OpenSubtitles part that was excluded.
author_association: COLLABORATOR
user: login: stefan-it
created_at: 2025-03-18T08:56:57Z
body: Thanks @alanakbik , current version looks really good, I am just wondering, if we really need empty tokens + `O` in the dataset:

![image](https://github.com/user-attachments/assets/b055acb3-cd8d-4652-bb54-53d9d9f00cf1)

I was able to avoid this by using this code fragment:

```
# Write tokens and tags
                            for token, tag in zip(tokens, tags):
                                token_text = text[token["start"] : token["end"]]  # Don't strip the token
                                if token_text:  # Still skip empty tokens
                                    # Replace newlines with space in output to maintain CoNLL format
                                    token_text = token_text.replace("\n", " ")

                                    if not token_text.strip():
                                        continue

                                    f_out.write(f"{token_text}\t{tag}\n")
```

:thinking: 
author_association: MEMBER
user: login: alanakbik
created_at: 2025-03-18T12:07:35Z
body: @stefan-it good point. One issue is that some "sentences" actually only consist of a single non-token, so there would need to be logic to handle this as well, otherwise we might get multiple empty lines after another. I considered filtering out those sentences, but then the dataset statistics no longer line up with the statistics posted on the DANSK page. Do you think it still makes sense to remove empty tokens and sentences, or should we prioritize matching the dataset statistics?
author_association: COLLABORATOR
user: login: stefan-it
created_at: 2025-03-19T07:34:02Z
body: So I think for reproducibility reasons, we should then stick to the current solution, to really match the numbers from their documentation ☺️
author_association: MEMBER