Repository: aphp/edsnlp (language: python)

Issue:
Problem with special whitespaces using `spacy.blank("eds")`
Edge cases with the `eds` tokenizer

## Description

Due to its more simple implementation, the `Tokenizer` shipped with the `eds` language doesn't handle every whitespace characters very well. For instance:

| Text   |      `spacy.blank('fr')`      |  `spacy.blank('eds')` |
|----------|:-------------:|------:|
| `"il\xa0fait\tchaud et froid"` |  `['il', '\xa0', 'fait', '\t', 'chaud', 'et', 'froid']` | `['il\xa0fait\tchaud', 'et', 'froid']` |

This is then especially problematic when using the Term Matcher 

## How to reproduce the bug

<!-- Add a minimal example that makes bug appears -->

```python
import spacy

langs = dict(
    fr=spacy.blank('fr'),
    eds=spacy.blank('eds'),
)

text = "il\xa0fait\tchaud et froid"

for lang, nlp in langs.items():
    doc = nlp(text)
    print(lang)
    print([repr(t) for t in doc])
```
## Suggestions

- Go back to the classic `FrenchTokenizer` but change some defaults to get the same features
- Handle the case in the `normalizer` pipeline
- Change the [`word_regex` ](https://github.com/aphp/edsnlp/blob/master/edsnlp/language.py#L64) to handle different types of whitespaces



Discussion / hints:

Fix patch:
diff --git a/changelog.md b/changelog.md
index 180774f0b..cf0b2222b 100644
--- a/changelog.md
+++ b/changelog.md
@@ -5,6 +5,7 @@
 ### Fixed
 
 - Add nephew, niece and daughter to family qualifier patterns
+- EDSTokenizer (`spacy.blank('eds')`) now recognizes non-breaking whitespaces as spaces and does not split float numbers
 
 ## v0.7.0 (2022-09-06)
 
diff --git a/docs/languages.md b/docs/languages.md
index 40fad65b7..f03a40961 100644
--- a/docs/languages.md
+++ b/docs/languages.md
@@ -8,10 +8,9 @@ the `fr` tokenizer. The main differences lie in the tokenization process.
 A comparison of the two tokenization methods is demonstrated below:
 
 | Example               | FrenchLanguage             | EDSLanguage                |
-| --------------------- | -------------------------- | -------------------------- |
+| --------------------- | -------------------------- |----------------------------|
 | `ACR 5`               | [`ACR5`]                   | [`ACR`, `5`]               |
-| `26.5`                | [`26.5`]                   | [`26`, `.`, `5`]           |
-| `26.5/`               | [`26.5/`]                  | [`26`, `.`, `5`, `/`]      |
+| `26.5/`               | [`26.5/`]                  | [`26.5`, `/`]              |
 | `\n \n CONCLUSION`    | [`\n \n`, `CONCLUSION]`    | [`\n`, `\n`, `CONCLUSION`] |
 | `l'artère`            | [`l'`, `artère`]           | [`l'`, `artère`] (same)    |
 
diff --git a/edsnlp/language.py b/edsnlp/language.py
index d34f2f8b8..26b5fa1d1 100644
--- a/edsnlp/language.py
+++ b/edsnlp/language.py
@@ -58,10 +58,10 @@ def __init__(self, vocab: Vocab) -> None:
         """
         self.vocab = vocab
         punct = "[:punct:]" + "\"'ˊ＂〃ײ᳓″״‶˶ʺ“”˝"
-        num_like = r"[\d]+"
-        default = rf"[^\d{punct}'\n ]+(?:['ˊ](?=[[:alpha:]]))?"
+        num_like = r"\d+(?:[.,]\d+)?"
+        default = rf"[^\d{punct}'\n[[:space:]]+(?:['ˊ](?=[[:alpha:]]))?"
         self.word_regex = regex.compile(
-            rf"({num_like}|[{punct}]|\n|[ ]+|{default})([ ])?"
+            rf"({num_like}|[{punct}]|[\n\r\t]|[^\S\r\n\t]+|{default})([^\S\r\n\t])?"
         )
 
     def __call__(self, text: str) -> Doc:


Test patch:
diff --git a/tests/test_language.py b/tests/test_language.py
index c96045869..5f569e94c 100644
--- a/tests/test_language.py
+++ b/tests/test_language.py
@@ -41,3 +41,29 @@ def test_eds_tokenizer_handles_long_text():
 def test_eds_lex_attrs_capitals(word):
     assert like_num(word)
     assert like_num(word.upper())
+
+
+def test_eds_tokenizer_whitespace():
+    nlp = spacy.blank("eds")
+    tokenized = [(w.text, w.whitespace_) for w in nlp("Lorem\xA0Ipsum\tDolor Sit Amet")]
+    assert tokenized == [
+        ("Lorem", " "),
+        ("Ipsum", ""),
+        ("\t", ""),
+        ("Dolor", " "),
+        ("Sit", " "),
+        ("Amet", ""),
+    ]
+
+
+def test_eds_tokenizer_numbers():
+    nlp = spacy.blank("eds")
+    tokenized = [(w.text, w.whitespace_) for w in nlp("Il fait 5.3/5.4mm")]
+    assert tokenized == [
+        ("Il", " "),
+        ("fait", " "),
+        ("5.3", ""),
+        ("/", ""),
+        ("5.4", ""),
+        ("mm", ""),
+    ]