Repository: flairNLP/flair

Issue:
Offset document boundaries in Dutch CoNLL-03 dataset
The Dutch CoNLL-03 NER dataset has a peculiar format in which document boundary tokens ("-DOCSTART-") are not given as separate sentences. Rather, they are given as first token of the first sentence of a document: 

~~~
-DOCSTART- -DOCSTART- O
Dat Pron O
is V O
verder Adj O
opgelaaid N O
door Prep O
windsnelheden N O
die Pron O
oplopen V O
tot Prep O
35 Num O
kilometer N O
per Prep O
uur N O
. Punc O
~~~

This confuses the Flair corpus object to believe that the -DOCSTART- token is the first word of this sentence. To fix this, newlines need to be added, so it becomes formatted like this: 

~~~
-DOCSTART- -DOCSTART- O

Dat Pron O
is V O
verder Adj O
opgelaaid N O
door Prep O
windsnelheden N O
die Pron O
oplopen V O
tot Prep O
35 Num O
kilometer N O
per Prep O
uur N O
. Punc O
~~~

This would also make it consistent with the other CoNLL-03 datasets (English and German). 

This PR tracks the progress of fixing this.


Discussion:

Resolving patch:
diff --git a/flair/datasets/sequence_labeling.py b/flair/datasets/sequence_labeling.py
--- a/flair/datasets/sequence_labeling.py
+++ b/flair/datasets/sequence_labeling.py
@@ -555,13 +555,25 @@ def __init__(
 
         # download data if necessary
         conll_02_path = "https://www.clips.uantwerpen.be/conll2002/ner/data/"
-        cached_path(f"{conll_02_path}ned.testa", Path("datasets") / dataset_name)
-        cached_path(f"{conll_02_path}ned.testb", Path("datasets") / dataset_name)
-        cached_path(f"{conll_02_path}ned.train", Path("datasets") / dataset_name)
+
+        # download files if not present locally
+        cached_path(f"{conll_02_path}ned.testa", data_folder / 'raw')
+        cached_path(f"{conll_02_path}ned.testb", data_folder / 'raw')
+        cached_path(f"{conll_02_path}ned.train", data_folder / 'raw')
+
+        # we need to slightly modify the original files by adding some new lines after document separators
+        train_data_file = data_folder / 'train.txt'
+        if not train_data_file.is_file():
+            self.__offset_docstarts(data_folder / 'raw' / "ned.train", data_folder / 'train.txt')
+            self.__offset_docstarts(data_folder / 'raw' / "ned.testa", data_folder / 'dev.txt')
+            self.__offset_docstarts(data_folder / 'raw' / "ned.testb", data_folder / 'test.txt')
 
         super(CONLL_03_DUTCH, self).__init__(
             data_folder,
             columns,
+            train_file='train.txt',
+            dev_file='dev.txt',
+            test_file='test.txt',
             tag_to_bioes=tag_to_bioes,
             encoding="latin-1",
             in_memory=in_memory,
@@ -569,6 +581,17 @@ def __init__(
             **corpusargs,
         )
 
+    @staticmethod
+    def __offset_docstarts(file_in: Union[str, Path], file_out: Union[str, Path]):
+        with open(file_in, 'r', encoding="latin-1") as f:
+            lines = f.readlines()
+        with open(file_out, 'w', encoding="latin-1") as f:
+            for line in lines:
+                f.write(line)
+                if line.startswith('-DOCSTART-'):
+                    f.write("\n")
+
+
 
 def add_IOB_tags(data_file: Union[str, Path], encoding: str = "utf8", ner_column: int = 1):
     """


Test patch: