Repository: flairNLP/flair

Issue:
HashTag ignored for Column Corpus
**Describe the bug**
When reading from a ColumnCorpus, lines with hash tags are ignored. This may bring problems to tasks such as Wnut17(there are hash tags at start of words.)
See https://github.com/zalandoresearch/flair/blob/b0e4cc1a92c4bd56536ed4e09c8ff6cb1a89f08c/flair/datasets.py#L424

**To Reproduce**
Run NER experiment with Wnut17.

**Expected behavior**
Hash-tag-starting words should be kept.

**Environment (please complete the following information):**
 - OS [e.g. iOS, Linux]: linux
 - Version [e.g. flair-0.3.2]: flair-0.4.2

**Additional context**
Add any other context about the problem here.



Discussion:
@ZihanWangKi thanks for spotting this - I can reproduce the error. Hash tags are used in CoNLL-U datasets to [denote comments](https://github.com/UniversalDependencies/docs/issues/642), which is why we currently ignore lines that begin with them. 

However, you're right that we need special rules for some datasets such as WNUT where a line that begins with a hash tag is just a word. Will put in a PR for this! 

Resolving patch:
diff --git a/flair/datasets.py b/flair/datasets.py
--- a/flair/datasets.py
+++ b/flair/datasets.py
@@ -26,6 +26,7 @@ def __init__(
         test_file=None,
         dev_file=None,
         tag_to_bioes=None,
+        comment_symbol: str = None,
         in_memory: bool = True,
     ):
         """
@@ -81,13 +82,21 @@ def __init__(
 
         # get train data
         train = ColumnDataset(
-            train_file, column_format, tag_to_bioes, in_memory=in_memory
+            train_file,
+            column_format,
+            tag_to_bioes,
+            comment_symbol=comment_symbol,
+            in_memory=in_memory,
         )
 
         # read in test file if exists, otherwise sample 10% of train data as test dataset
         if test_file is not None:
             test = ColumnDataset(
-                test_file, column_format, tag_to_bioes, in_memory=in_memory
+                test_file,
+                column_format,
+                tag_to_bioes,
+                comment_symbol=comment_symbol,
+                in_memory=in_memory,
             )
         else:
             train_length = len(train)
@@ -99,7 +108,11 @@ def __init__(
         # read in dev file if exists, otherwise sample 10% of train data as dev dataset
         if dev_file is not None:
             dev = ColumnDataset(
-                dev_file, column_format, tag_to_bioes, in_memory=in_memory
+                dev_file,
+                column_format,
+                tag_to_bioes,
+                comment_symbol=comment_symbol,
+                in_memory=in_memory,
             )
         else:
             train_length = len(train)
@@ -377,12 +390,14 @@ def __init__(
         path_to_column_file: Path,
         column_name_map: Dict[int, str],
         tag_to_bioes: str = None,
+        comment_symbol: str = None,
         in_memory: bool = True,
     ):
         assert path_to_column_file.exists()
         self.path_to_column_file = path_to_column_file
         self.tag_to_bioes = tag_to_bioes
         self.column_name_map = column_name_map
+        self.comment_symbol = comment_symbol
 
         # store either Sentence objects in memory, or only file offsets
         self.in_memory = in_memory
@@ -421,7 +436,7 @@ def __init__(
 
             while line:
 
-                if line.startswith("#"):
+                if self.comment_symbol is not None and line.startswith(comment_symbol):
                     line = f.readline()
                     continue
 
@@ -479,7 +494,7 @@ def __getitem__(self, index: int = 0) -> Sentence:
                 line = file.readline()
                 sentence: Sentence = Sentence()
                 while line:
-                    if line.startswith("#"):
+                    if self.comment_symbol is not None and line.startswith("#"):
                         line = file.readline()
                         continue
 
@@ -1055,7 +1070,11 @@ def __init__(
             )
             log.warning("-" * 100)
         super(GERMEVAL, self).__init__(
-            data_folder, columns, tag_to_bioes=tag_to_bioes, in_memory=in_memory
+            data_folder,
+            columns,
+            tag_to_bioes=tag_to_bioes,
+            comment_symbol="#",
+            in_memory=in_memory,
         )
 
 


Test patch: