Repository: fnl/syntok

Issue:
Bug in not-contraction handling code
Hi,

Thanks for the great library! I think I ran into a weird edge-case wrt not-contraction handling code. If I use the following example:
```python
from syntok.tokenizer import Tokenizer

tok = Tokenizer(replace_not_contraction=False)
tok.split("n't")
```

The output is `[<Token '' : "n't" @ 1>]`. Something is going wrong in the offset calculation there, that 1 should be a 0... The real example this came from is a sentence in the AIDA dataset, `" Falling share prices in New York do n't hurt Mexico as long as it happens gradually , as earlier this week`.

I see the same with "don't": `[<Token '' : 'do' @ 0>, <Token '' : "n't" @ 3>]`, that 3 should be 2 no?

Would love to hear your thoughts, not sure how to fix this neatly yet.

Discussion:
fnl: I think offsets should all ways be truthful to the input text, as that is where there value lies: tracking the provenance of a token.

So even can’t would become “ca”:0, “not”:2 in that scenario.

Does that seem right to you, too?
KDercksen: Yes, I agree actually! I was doubting myself because of the comment written [here](https://github.com/fnl/syntok/blob/master/syntok/segmenter.py#L37) though.

I do think it would be best, even if the token text changes ("n't" -> "not"), that the offsets should be truthful to the original text. I can change the PR to reflect this!

Patch:
diff --git a/syntok/tokenizer.py b/syntok/tokenizer.py
index e27a705..edb3f7f 100644
--- a/syntok/tokenizer.py
+++ b/syntok/tokenizer.py
@@ -221,7 +221,7 @@ class Tokenizer:
                     yield Token(prefix, word[remainder:mo.start() - 1], offset + remainder)
                     prefix = ""
 
-                yield Token(prefix, "not" if self.replace_not_contraction else 'n' + mo.group(0), offset + mo.start())
+                yield Token(prefix, "not" if self.replace_not_contraction else 'n' + mo.group(0), offset + mo.start() - 1)
                 return ""
 
             yield Token(prefix, word[remainder:mo.start()], offset + remainder)


Test patch:
diff --git a/syntok/tokenizer_test.py b/syntok/tokenizer_test.py
index e40a032..dbb3cbe 100644
--- a/syntok/tokenizer_test.py
+++ b/syntok/tokenizer_test.py
@@ -91,6 +91,23 @@ class TestTokenizer(TestCase):
         self.assertListEqual(s(result), ["\U0001F64C", ".", "A"])
         self.assertListEqual([t.offset for t in result], [0, 1, 2])  # requires Py3.3+
 
+    def test_apostrophe_offset_without_replace_not_contraction(self):
+        # NOTE: in this case nothing is replaced, so the offsets should remain identical
+        # to those in the original text
+        text = "don't"
+        self.tokenizer = Tokenizer(replace_not_contraction=False)
+        result = self.tokenizer.split(text)
+        self.assertListEqual([t.offset for t in result], [0, 2])
+
+    def test_apostrophe_offset_with_replace_not_contraction(self):
+        # NOTE: in this case, "n't" is replaced with "not", so a space is introduced.
+        # e.g. "don't" -> "do not", "can't" -> "can not"
+        text = "don't"
+        self.tokenizer = Tokenizer(replace_not_contraction=True)
+        result = self.tokenizer.split(text)
+        self.assertListEqual([t.offset for t in result], [0, 2])
+        self.assertListEqual([t.value for t in result], ["do", "not"])
+
 
 class TestToText(TestCase):