Repository: nltk/nltk

Issue:
span_tokenize failed when sentence contains double quotation
If we feed in a sentence with double quotation into TreebankWordTokenizer's span_tokenize function, there will be errors. Probably this is because the function sends the raw string input along with the tokenized string to the align_tokens function, without considering that the tokenize function would replace double quotation marks with something else.
TreebankWordDetokenizer is not always an inverse of TreebankWordTokenizer
For example, for the following script:

```python

import nltk
import nltk.tokenize

tokens = nltk.tokenize.TreebankWordTokenizer().tokenize("I wanna watch something")
print(tokens)
sentence = nltk.tokenize.treebank.TreebankWordDetokenizer().detokenize(tokens)
print(sentence)
```

The result is:
```
['I', 'wan', 'na', 'watch', 'something']
I wannawatch something
```

The detokenized sentence should be: `I wanna watch something`


Discussion:
Thanks @albertauyeung for reporting the issue. Do you have an example where you met an error with the `TreebankWordTokenizer.span_tokenize()`?

Do you mean something like this?

```python
>>> from nltk.tokenize.treebank import TreebankWordTokenizer
>>> tbw = TreebankWordTokenizer
>>> tbw = TreebankWordTokenizer()
>>> s = '''This is a sentence with "quotes inside" and alsom some 'single quotes', etc.'''
>>> print(s)
This is a sentence with "quotes inside" and alsom some 'single quotes', etc.
>>> tbw.span_tokenize(s)
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/nltk/tokenize/util.py", line 230, in align_tokens
    start = sentence.index(token, point)
ValueError: substring not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.5/site-packages/nltk/tokenize/treebank.py", line 167, in span_tokenize
    return align_tokens(tokens, text)
  File "/usr/local/lib/python3.5/site-packages/nltk/tokenize/util.py", line 232, in align_tokens
    raise ValueError('substring "{}" not found in "{}"'.format(token, sentence))
ValueError: substring "``" not found in "This is a sentence with "quotes inside" and alsom some 'single quotes', etc."
```

**Suboptimal solution**:

```python
>>> s = '''This is a sentence with `` quotes inside '' and alsom some 'single quotes', etc.''' 
>>> tbw.span_tokenize(s)
[(0, 4), (5, 7), (8, 9), (10, 18), (19, 23), (24, 26), (27, 33), (34, 40), (41, 43), (44, 47), (48, 53), (54, 58), (59, 66), (67, 73), (73, 74), (74, 75), (76, 79), (79, 80)]
```
@alvations Yes. That's the exact error I got. Right now it seems we have to preprocess the sentence before submitting to span_tokenize.
A simple solution would be to replace the quotes before calling the `nltk.tokenize.util.align_tokens` function at https://github.com/nltk/nltk/blob/develop/nltk/tokenize/treebank.py#L147

```python
    def span_tokenize(self, text):
        tokens = self.tokenize(text)
        tokens = ['"' if tok in ['``', "''"] else tok for tok in tokens]
        return align_tokens(tokens, text)
```

After the patch:

```python
>>> from nltk.tokenize.treebank import TreebankWordTokenizer
>>> tbw = TreebankWordTokenizer()
>>> s = '''This is a sentence with "quotes inside" and alsom some 'single quotes', etc.'''
>>> print(s)
This is a sentence with "quotes inside" and alsom some 'single quotes', etc.
>>> tbw.span_tokenize(s)
[(0, 4), (5, 7), (8, 9), (10, 18), (19, 23), (24, 25), (25, 31), (32, 38), (38, 39), (40, 43), (44, 49), (50, 54), (55, 62), (63, 69), (69, 70), (70, 71), (72, 75), (75, 76)]
```


@albertauyeung do you want to take a stab at adding the patch and create a new pull-request?
@alvations Yes, sure. Will do! 
Fixed on #1751 
Please note that this fix will still throw an exception for a text with both types of quotes:
nltk.TreebankWordTokenizer().span_tokenize('" ``')
Hi @alyaxey, what is the exception you see?

I executed `nltk.TreebankWordTokenizer().span_tokenize('" ``')` and got the following:
`[(0, 1), (2, 4)]`

Sorry, I've provided a wrong test case. Please take a look at this one:
```
import nltk
print(nltk.TreebankWordTokenizer().span_tokenize('``` "'))
```
The expected output is `[(0, 2), (2, 3), (4, 5)]` if we follow the logic of the current tokenize method. Also `[(0, 3), (4, 5)]` is acceptable.
Here's my output for developer branch:
```
Traceback (most recent call last):
  File "/Users/alyaxey/Downloads/nltk-develop/nltk/tokenize/util.py", line 254, in align_tokens
    start = sentence.index(token, point)
ValueError: substring not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 2, in <module>
    print(nltk.TreebankWordTokenizer().span_tokenize('``` "'))
  File "/Users/alyaxey/Downloads/nltk-develop/nltk/tokenize/treebank.py", line 179, in span_tokenize
    return align_tokens(tokens, text)
  File "/Users/alyaxey/Downloads/nltk-develop/nltk/tokenize/util.py", line 256, in align_tokens
    raise ValueError('substring "{}" not found in "{}"'.format(token, sentence))
ValueError: substring "`" not found in "``` ""
```
I'd like to suggest a different solution to 1) fix this and similar bugs, 2) provide more flexibility to users, 3) make code more clear. We can add a boolean parameter to `tokenize` method that enables or disables quotes transformation. We can disable quotes transformation during span_tokenize to avoid any non-space manipulations.
I am running into an exception with the current version of span_tokenize for strings that contain brackets before quotation marks. I believe the regex is wrong since it also matches brackets and later replaces quotation marks in the "raw_tokens" with those brackets. Or am I missing something?

**Example:** 
``` python
s = ' ( see 6)  Biotin " " affinity'
w_spans = TreebankWordTokenizer().span_tokenize(s) 
```

**Exception:** 
```
...
  File "/home/mp/miniconda3/envs/py36/lib/python3.6/site-packages/nltk/tokenize/treebank.py", line 179, in span_tokenize
    return align_tokens(tokens, text)
  File "/home/mp/miniconda3/envs/py36/lib/python3.6/site-packages/nltk/tokenize/util.py", line 256, in align_tokens
    raise ValueError('substring "{}" not found in "{}"'.format(token, sentence))
ValueError: substring "(" not found in " ( see 6)  Biotin " " affinity"
```

**Suggested fix:** 
Change the regex in span_tokenize from `r'[(``)(\'\')(")]+'` to `r'(``)|(\'\')|(")'`
Ok my bad, it has actually already been fixed in commit 4b21300999e11ba6f91952c05a936ccec0673e2e and is running like a charm in nltk-3.3
oh, it stiil has problem in nltk-3.3

like this:

```
File "/home/users/----/.miniconda2/lib/python2.7/site-packages/nltk/tokenize/util.py", line 258, in align_tokens
    raise ValueError('substring "{}" not found in "{}"'.format(token, sentence))
ValueError: substring "''" not found in "''Elton's been through a lot," he told The Sun newspaper."

```
@memeda I confirm I can reproduce this bug. A solution is to add one more regex to match single quotes at the beginning of a string. Please see my branch on https://github.com/albertauyeung/nltk/tree/hotfix-span-tokenizer
Confirmed:

```
raise ValueError('substring "{}" not found in "{}"'.format(token, sentence))
```
```
ValueError: substring "enriched" not found in "The Hindu describing his Cricket, once said: `` His batting resembles very closely that of his father -dashing and carefree -and his cover-drive, a joy to watch, has amazing impetus...''And it added that he had ``enriched Madras sport as his father had''."
```
Hi, I also experienced this bug, e. g. it happens on the following text:
```
''Cosita Linda' - Lisandro (2013)\n\"El Clon (2010) .... Mohammed
```

The resulting error is as follows:

```
ValueError: substring "''" not found in "''Cosita Linda' - Lisandro (2013)
"El Clon (2010) .... Mohammed"
```

Are there any updates on this problem?

Experienced this same crash today. Reproductible example:
```
from nltk import TreebankWordTokenizer
tokenizer = TreebankWordTokenizer()
list(tokenizer.span_tokenize('\'\'Y\'"'))
```
yields 
`ValueError: substring "''" not found in "''Y'""`
Hello!

For context, this comment might help:
https://github.com/nltk/nltk/blob/f12df18465ba642d93f8f2f5d9a85147a2e1a450/nltk/tokenize/destructive.py#L34-L42

In short, TreebankWordDetokenizer is not *guaranteed* to revert to the original string. This is because there are multiple inputs that lead to the same output - which make it impossible to know which of the multiple inputs is desired.

*That said*, there definitely seems to be a bug when looking at your example - it should have been able to know that "wanna" and "watch" should not become one word. 

Resolving patch:
diff --git a/nltk/tokenize/__init__.py b/nltk/tokenize/__init__.py
--- a/nltk/tokenize/__init__.py
+++ b/nltk/tokenize/__init__.py
@@ -88,7 +88,7 @@
 from nltk.tokenize.stanford_segmenter import StanfordSegmenter
 from nltk.tokenize.texttiling import TextTilingTokenizer
 from nltk.tokenize.toktok import ToktokTokenizer
-from nltk.tokenize.treebank import TreebankWordTokenizer
+from nltk.tokenize.treebank import TreebankWordDetokenizer, TreebankWordTokenizer
 from nltk.tokenize.util import regexp_span_tokenize, string_span_tokenize
 
 
diff --git a/nltk/tokenize/destructive.py b/nltk/tokenize/destructive.py
--- a/nltk/tokenize/destructive.py
+++ b/nltk/tokenize/destructive.py
@@ -1,7 +1,8 @@
 # Natural Language Toolkit: NLTK's very own tokenizer.
 #
 # Copyright (C) 2001-2021 NLTK Project
-# Author:
+# Author: Liling Tan
+#         Tom Aarsen <> (modifications)
 # URL: <https://www.nltk.org>
 # For license information, see LICENSE.TXT
 
@@ -9,6 +10,7 @@
 import re
 
 from nltk.tokenize.api import TokenizerI
+from nltk.tokenize.util import align_tokens
 
 
 class MacIntyreContractions:
@@ -24,7 +26,7 @@ class MacIntyreContractions:
         r"(?i)\b(got)(?#X)(ta)\b",
         r"(?i)\b(lem)(?#X)(me)\b",
         r"(?i)\b(more)(?#X)('n)\b",
-        r"(?i)\b(wan)(?#X)(na)\s",
+        r"(?i)\b(wan)(?#X)(na)(?=\s)",
     ]
     CONTRACTIONS3 = [r"(?i) ('t)(?#X)(is)\b", r"(?i) ('t)(?#X)(was)\b"]
     CONTRACTIONS4 = [r"(?i)\b(whad)(dd)(ya)\b", r"(?i)\b(wha)(t)(cha)\b"]
@@ -53,8 +55,8 @@ class NLTKWordTokenizer(TokenizerI):
     # Ending quotes.
     ENDING_QUOTES = [
         (re.compile("([»”’])", re.U), r" \1 "),
+        (re.compile(r"''"), " '' "),
         (re.compile(r'"'), " '' "),
-        (re.compile(r"(\S)(\'\')"), r"\1 \2 "),
         (re.compile(r"([^' ])('[sS]|'[mM]|'[dD]|') "), r"\1 \2 "),
         (re.compile(r"([^' ])('ll|'LL|'re|'RE|'ve|'VE|n't|N'T) "), r"\1 \2 "),
     ]
@@ -146,3 +148,55 @@ def tokenize(self, text, convert_parentheses=False, return_str=False):
         #     text = regexp.sub(r' \1 \2 \3 ', text)
 
         return text if return_str else text.split()
+
+    def span_tokenize(self, text):
+        r"""
+        Uses the post-hoc nltk.tokens.align_tokens to return the offset spans.
+            >>> from nltk.tokenize import NLTKWordTokenizer
+            >>> s = '''Good muffins cost $3.88\nin New (York).  Please (buy) me\ntwo of them.\n(Thanks).'''
+            >>> expected = [(0, 4), (5, 12), (13, 17), (18, 19), (19, 23),
+            ... (24, 26), (27, 30), (31, 32), (32, 36), (36, 37), (37, 38),
+            ... (40, 46), (47, 48), (48, 51), (51, 52), (53, 55), (56, 59),
+            ... (60, 62), (63, 68), (69, 70), (70, 76), (76, 77), (77, 78)]
+            >>> list(NLTKWordTokenizer().span_tokenize(s)) == expected
+            True
+            >>> expected = ['Good', 'muffins', 'cost', '$', '3.88', 'in',
+            ... 'New', '(', 'York', ')', '.', 'Please', '(', 'buy', ')',
+            ... 'me', 'two', 'of', 'them.', '(', 'Thanks', ')', '.']
+            >>> [s[start:end] for start, end in NLTKWordTokenizer().span_tokenize(s)] == expected
+            True
+            Additional example
+            >>> from nltk.tokenize import NLTKWordTokenizer
+            >>> s = '''I said, "I'd like to buy some ''good muffins" which cost $3.88\n each in New (York)."'''
+            >>> expected = [(0, 1), (2, 6), (6, 7), (8, 9), (9, 10), (10, 12),
+            ... (13, 17), (18, 20), (21, 24), (25, 29), (30, 32), (32, 36),
+            ... (37, 44), (44, 45), (46, 51), (52, 56), (57, 58), (58, 62),
+            ... (64, 68), (69, 71), (72, 75), (76, 77), (77, 81), (81, 82),
+            ... (82, 83), (83, 84)]
+            >>> list(NLTKWordTokenizer().span_tokenize(s)) == expected
+            True
+            >>> expected = ['I', 'said', ',', '"', 'I', "'d", 'like', 'to',
+            ... 'buy', 'some', "''", "good", 'muffins', '"', 'which', 'cost',
+            ... '$', '3.88', 'each', 'in', 'New', '(', 'York', ')', '.', '"']
+            >>> [s[start:end] for start, end in NLTKWordTokenizer().span_tokenize(s)] == expected
+            True
+        """
+        raw_tokens = self.tokenize(text)
+
+        # Convert converted quotes back to original double quotes
+        # Do this only if original text contains double quote(s) or double
+        # single-quotes (because '' might be transformed to `` if it is
+        # treated as starting quotes).
+        if ('"' in text) or ("''" in text):
+            # Find double quotes and converted quotes
+            matched = [m.group() for m in re.finditer(r"``|'{2}|\"", text)]
+
+            # Replace converted quotes back to double quotes
+            tokens = [
+                matched.pop(0) if tok in ['"', "``", "''"] else tok
+                for tok in raw_tokens
+            ]
+        else:
+            tokens = raw_tokens
+
+        yield from align_tokens(tokens, text)
diff --git a/nltk/tokenize/treebank.py b/nltk/tokenize/treebank.py
--- a/nltk/tokenize/treebank.py
+++ b/nltk/tokenize/treebank.py
@@ -3,6 +3,7 @@
 # Copyright (C) 2001-2021 NLTK Project
 # Author: Edward Loper <edloper@gmail.com>
 #         Michael Heilman <mheilman@cmu.edu> (re-port from http://www.cis.upenn.edu/~treebank/tokenizer.sed)
+#         Tom Aarsen <> (modifications)
 #
 # URL: <https://www.nltk.org>
 # For license information, see LICENSE.TXT
@@ -86,8 +87,8 @@ class TreebankWordTokenizer(TokenizerI):
 
     # ending quotes
     ENDING_QUOTES = [
+        (re.compile(r"''"), " '' "),
         (re.compile(r'"'), " '' "),
-        (re.compile(r"(\S)(\'\')"), r"\1 \2 "),
         (re.compile(r"([^' ])('[sS]|'[mM]|'[dD]|') "), r"\1 \2 "),
         (re.compile(r"([^' ])('ll|'LL|'re|'RE|'ve|'VE|n't|N'T) "), r"\1 \2 "),
     ]


Test patch:
diff --git a/nltk/test/tokenize.doctest b/nltk/test/tokenize.doctest
--- a/nltk/test/tokenize.doctest
+++ b/nltk/test/tokenize.doctest
@@ -3,10 +3,10 @@
 
     >>> from nltk.tokenize import *
 
-Regression Tests: Treebank Tokenizer
+Regression Tests: NLTKWordTokenizer
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Some test strings.
+Tokenizing some test strings.
 
     >>> s1 = "On a $50,000 mortgage of 30 years at 8 percent, the monthly payment would be $366.88."
     >>> word_tokenize(s1)
@@ -42,6 +42,35 @@ Some test strings.
     >>> word_tokenize(s11)
     ['It', "'s", 'more', "'n", 'enough', '.']
 
+Gathering the spans of the tokenized strings.
+
+    >>> s = '''Good muffins cost $3.88\nin New (York).  Please (buy) me\ntwo of them.\n(Thanks).'''
+    >>> expected = [(0, 4), (5, 12), (13, 17), (18, 19), (19, 23),
+    ... (24, 26), (27, 30), (31, 32), (32, 36), (36, 37), (37, 38),
+    ... (40, 46), (47, 48), (48, 51), (51, 52), (53, 55), (56, 59),
+    ... (60, 62), (63, 68), (69, 70), (70, 76), (76, 77), (77, 78)]
+    >>> list(NLTKWordTokenizer().span_tokenize(s)) == expected
+    True
+    >>> expected = ['Good', 'muffins', 'cost', '$', '3.88', 'in',
+    ... 'New', '(', 'York', ')', '.', 'Please', '(', 'buy', ')',
+    ... 'me', 'two', 'of', 'them.', '(', 'Thanks', ')', '.']
+    >>> [s[start:end] for start, end in NLTKWordTokenizer().span_tokenize(s)] == expected
+    True
+
+    >>> s = '''I said, "I'd like to buy some ''good muffins" which cost $3.88\n each in New (York)."'''
+    >>> expected = [(0, 1), (2, 6), (6, 7), (8, 9), (9, 10), (10, 12),
+    ... (13, 17), (18, 20), (21, 24), (25, 29), (30, 32), (32, 36),
+    ... (37, 44), (44, 45), (46, 51), (52, 56), (57, 58), (58, 62),
+    ... (64, 68), (69, 71), (72, 75), (76, 77), (77, 81), (81, 82),
+    ... (82, 83), (83, 84)]
+    >>> list(NLTKWordTokenizer().span_tokenize(s)) == expected
+    True
+    >>> expected = ['I', 'said', ',', '"', 'I', "'d", 'like', 'to',
+    ... 'buy', 'some', "''", "good", 'muffins', '"', 'which', 'cost',
+    ... '$', '3.88', 'each', 'in', 'New', '(', 'York', ')', '.', '"']
+    >>> [s[start:end] for start, end in NLTKWordTokenizer().span_tokenize(s)] == expected
+    True
+
 Testing improvement made to the TreebankWordTokenizer
 
     >>> sx1 = '\xabNow that I can do.\xbb'