Repository: flairNLP/flair

Issue:
Get original text after tokenizer in flair 0.11
Hi guys! I am rewrting code that previously used flair 0.10 to flair 0.11 and I'm running into some issues. It originally took an OCR phrase, instantiated a Sentence object, and predicted a topic using a torch model. So far everything perfect in both versions.

To see the different topics of the sentence, in flair 0.10 I had to access the "entities" while in 0.11 I access the "labels" (which in 0.10 appear as empty). The main problemthat I am not able to solve is that in flair 0.10 in each entity I could access not only the tokenized text but also the original text, which I am unable to do in flair 0.11. Can someone tell me if there is any way to do it?


Discussion:
Hello @rafgonlop1 thanks for pointing this out. 

If I understand correctly, this snippet reproduces your problem:

```python
model = SequenceTagger.load('ner-fast')

sentence = Sentence("George   Washington was here")

model.predict(sentence)

for entity in sentence.get_spans('ner'):
    print(entity)
    print(entity.text)
```

Where `print(entity.text)` prints `George Washington` instead of <code>George&nbsp;&nbsp;&nbsp;Washington</code>(with extra whitespace). 

Is this correct? If so, we could put in PR that fixes this so that `entity.text` will print out the original text instead of the joined tokens.

Hello @alanakbik. That's exactly my problem. In my case I use `sentence.get_labels()` but it's the same.  `sentence` object has the method `to_original_text()` along with attribute `text`. It would be great if the `Label` class also had both.
Ok! Could you modify my snippet to illustrate how you're currently using `get_labels()` and where you'd like a `to_original_text` method?
Of course, @alanakbik. I'm doing something like this:

```python
model = SequenceTagger.load('flair')

sentence = Sentence("ANDANA LAND CORPORA1@N PTY LTD SELLS HOUSES")

model.predict(sentence)

for entity in sentence.get_labels():
    print(entity) 
    print(entity.text)
```

In this case, `entity.text` is `'ANDANA LAND CORPORA1 @ N PTY LTD` and the orginial text would be `'ANDANA LAND CORPORA1@N PTY LTD`
Thanks! Is `entity.text` printing something for you here? The Label class should not have a text field.
You're absolutely correct. I meant `entity.data_point.text`. I think I'm understanding more the way you proceeded before. 
Got it, thanks! Will put in a PR!
Thank you @alanakbik, I tried to use just the same method flair has in Sentence class and works perfectly in the data_point :)

Resolving patch:
diff --git a/flair/data.py b/flair/data.py
--- a/flair/data.py
+++ b/flair/data.py
@@ -567,7 +567,7 @@ def end_position(self) -> int:
 
     @property
     def text(self) -> str:
-        return " ".join([t.text for t in self.tokens])
+        return "".join([t.text + t.whitespace_after * " " for t in self.tokens]).strip()
 
     @property
     def unlabeled_identifier(self) -> str:
@@ -748,7 +748,7 @@ def __init__(
 
     @property
     def unlabeled_identifier(self):
-        return f'Sentence: "{self.to_tokenized_string()}"'
+        return f'Sentence[{len(self)}]: "{self.text}"'
 
     def get_relations(self, type: str) -> List[Relation]:
         relations: List[Relation] = []
@@ -882,7 +882,7 @@ def to_tagged_string(self, main_label=None) -> str:
 
     @property
     def text(self):
-        return self.to_tokenized_string()
+        return "".join([t.text + t.whitespace_after * " " for t in self.tokens])
 
     def to_tokenized_string(self) -> str:
 
diff --git a/flair/file_utils.py b/flair/file_utils.py
--- a/flair/file_utils.py
+++ b/flair/file_utils.py
@@ -24,9 +24,9 @@
 logger = logging.getLogger("flair")
 
 
-def load_big_file(f: str) -> mmap.mmap:
+def load_big_file(f: str):
     """
-    Workaround for loading a big pickle file. Files over 2GB cause pickle errors on certin Mac and Windows distributions.
+    Workaround for loading a big pickle file. Files over 2GB cause pickle errors on certain Mac and Windows distributions.
     :param f:
     :return:
     """


Test patch: