Repository: nltk/nltk

Issue:
word_tokenizer and Spanish
I would like to have a word_tokenizer that works with Spanish. For example, this code:

`import nltk
from nltk.tokenize import word_tokenize
sentences = "¿Quién eres tú? ¡Hola! ¿Dónde estoy?"
spanish_sentence_tokenizer = nltk.data.load('tokenizers/punkt/spanish.pickle')
sentences = spanish_sentence_tokenizer.tokenize(sentences)
for s in sentences:
    print([s for s in vword_tokenize(s)])`

gives the following:

`['¿Quién', 'eres', 'tú', '?']
['¡Hola', '!']
['¿Dónde', 'estoy', '?']`

 but I would have expected the following instead:

`['¿' ,'Quién', 'eres', 'tú', '?']
['¡' ,'Hola', '!']
['¿' ,'Dónde', 'estoy', '?']`
Small improvement to the ToktokTokenizer
From #1558, we can easily add the `u'\xa1'` character to the `FUNKY_PUNCT_1` regular expression at https://github.com/nltk/nltk/blob/develop/nltk/tokenize/toktok.py#L51 to cover the inverted exclamation. 

It's an easy improvement to make as a first contribution to NLTK. You can easily add to the examples from #1558 to the doctest at https://github.com/nltk/nltk/blob/develop/nltk/tokenize/toktok.py#L45 to check whether it tokenize the string successfully `¡Hola!` -> `['¡' , 'Hola', '!']`

Please do take a look at https://github.com/nltk/nltk/blob/develop/CONTRIBUTING.md before contributing. 

----

Here's a quick cheatsheet to contributing:

 - First, fork the NLTK repository at https://github.com/nltk/nltk (Click on the "Fork" button on the top right)

 - Then go to your fork of NLTK and do a git clone to your machine/laptop, e.g. if your username on github is \<newcontributor\>, you should do something like `git clone https://github.com/<newcontributor>/nltk.git` on the terminal/command line or copy the `.git` address to the git GUI.

 - Follow the instructions from https://github.com/nltk/nltk/blob/develop/CONTRIBUTING.md#forks--github-pull-requests to checkout a new branch

 - Now navigate to the file in `nltk/nltk/tokenize/toktok.py` on your local machine. Add to the line https://github.com/nltk/nltk/blob/develop/nltk/tokenize/toktok.py#L51

 - Then write a test example in the docstring to show that the feature you added works (see https://github.com/nltk/nltk/blob/develop/nltk/tokenize/toktok.py#L45 )

 - Then validate that the tests you wrote work, something along the lines of `python -c doctest nltk/tokenize/toktok.py`, check it for `python3` too. For more information on doctest, see https://pymotw.com/2/doctest/

 - Now git add and commit your changes and update your fork of NLTK

 - Finally, go back to your github.com fork of the NLTK repository, and do a new pull request. 

Detailed instructions to do a git add, commit and doing a pull request can be found on https://github.com/nltk/nltk/blob/develop/CONTRIBUTING.md#forks--github-pull-requests


Discussion:
C.f. #1214, there are quite a few alternative tokenizers in NLTK =)

E.g. using NLTK port of @jonsafari [toktok tokenizer](https://github.com/jonsafari/tok-tok):

```python
>>> import nltk
>>> nltk.download('perluniprops')
[nltk_data] Downloading package perluniprops to
[nltk_data]     /Users/liling.tan/nltk_data...
[nltk_data]   Package perluniprops is already up-to-date!
True
>>> nltk.download('nonbreaking_prefixes')
[nltk_data] Downloading package nonbreaking_prefixes to
[nltk_data]     /Users/liling.tan/nltk_data...
[nltk_data]   Package nonbreaking_prefixes is already up-to-date!
True
>>> from nltk.tokenize.toktok import ToktokTokenizer
>>> toktok = ToktokTokenizer()
>>> sent = u"¿Quién eres tú? ¡Hola! ¿Dónde estoy?"
>>> toktok.tokenize(sent)
[u'\xbf', u'Qui\xe9n', u'eres', u't\xfa', u'?', u'\xa1Hola', u'!', u'\xbf', u'D\xf3nde', u'estoy', u'?']
>>> print " ".join(toktok.tokenize(sent))
¿ Quién eres tú ? ¡Hola ! ¿ Dónde estoy ?

>>> from nltk import sent_tokenize
>>> sentences = u"¿Quién eres tú? ¡Hola! ¿Dónde estoy?"
>>> [toktok.tokenize(sent) for sent in sent_tokenize(sentences, language='spanish')]
[[u'\xbf', u'Qui\xe9n', u'eres', u't\xfa', u'?'], [u'\xa1Hola', u'!'], [u'\xbf', u'D\xf3nde', u'estoy', u'?']]

>>> print '\n'.join([' '.join(toktok.tokenize(sent)) for sent in sent_tokenize(sentences, language='spanish')])
¿ Quién eres tú ?
¡Hola !
¿ Dónde estoy ?
```

If you hack the code a little and add `u'\xa1'` in https://github.com/nltk/nltk/blob/develop/nltk/tokenize/toktok.py#L51 , you should be able to get:

```
[[u'\xbf', u'Qui\xe9n', u'eres', u't\xfa', u'?'], [u'\xa1', u'Hola', u'!'], [u'\xbf', u'D\xf3nde', u'estoy', u'?']]
```
Thanks for your quick reply. I did as you directed but toktok isn't installed. What else do I need to do?

`>>> import nltk
nltk.download('perluniprops')
>>> nltk.download('perluniprops')
[nltk_data] Downloading package perluniprops to
[nltk_data]     /Users/barry/nltk_data...
[nltk_data]   Package perluniprops is already up-to-date!
True
>>> nltk.download('nonbreaking_prefixes')
[nltk_data] Downloading package nonbreaking_prefixes to
[nltk_data]     /Users/barry/nltk_data...
[nltk_data]   Package nonbreaking_prefixes is already up-to-date!
True
>>> from nltk.tokenize.toktok import ToktokTokenizer
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named toktok`
Please update your NLTK (but i think the toktok tokenizer is not in the official release, i.e. master branch yet).

```
pip install -U nltk
```

Please wait for the next version release to use the toktok Tokenizer. 

----

Meanwhile, if you urgently need to use the toktok tokenizer, it should reside in the development branch :

```
pip install -U https://github.com/nltk/nltk/archive/develop.zip
```

Note that the development branch is not an official release, it is subjected to unattested bugs. If you just need the ToktokTokenizer, you can get it directly from the source file at https://github.com/nltk/nltk/blob/develop/nltk/tokenize/toktok.py (it should be quite independent from NLTK, since it's made up of some clean regex substitutions)

Ok thanks! When I use this tokeniser I still get unexpected behaviour:

> ['¿', 'Quién', 'eres', 'tú', '?']
> ['¡Hola', '!']
> ['¿', 'Dónde', 'estoy', '?']

Notice '¡Hola'. This is an issue with toktok.pl. I have reported an issue here: https://github.com/jonsafari/tok-tok/issues/2
Please look at #1559 and jonsafari/tok-tok#1 .


Resolving patch:
diff --git a/nltk/tokenize/toktok.py b/nltk/tokenize/toktok.py
--- a/nltk/tokenize/toktok.py
+++ b/nltk/tokenize/toktok.py
@@ -38,17 +38,17 @@ class ToktokTokenizer(TokenizerI):
     >>> text = u'The https://github.com/jonsafari/tok-tok/blob/master/tok-tok.pl is a website with/and/or slashes and sort of weird : things'
     >>> print (toktok.tokenize(text, return_str=True))
     The https://github.com/jonsafari/tok-tok/blob/master/tok-tok.pl is a website with/and/or slashes and sort of weird : things
-    >>> text = u'This, is a sentence with weird\xbb symbols\u2026 appearing everywhere\xbf'
-    >>> expected = u'This , is a sentence with weird \xbb symbols \u2026 appearing everywhere \xbf'
+    >>> text = u'\xa1This, is a sentence with weird\xbb symbols\u2026 appearing everywhere\xbf'
+    >>> expected = u'\xa1 This , is a sentence with weird \xbb symbols \u2026 appearing everywhere \xbf'
     >>> assert toktok.tokenize(text, return_str=True) == expected
-    >>> toktok.tokenize(text) == [u'This', u',', u'is', u'a', u'sentence', u'with', u'weird', u'\xbb', u'symbols', u'\u2026', u'appearing', u'everywhere', u'\xbf']
+    >>> toktok.tokenize(text) == [u'\xa1', u'This', u',', u'is', u'a', u'sentence', u'with', u'weird', u'\xbb', u'symbols', u'\u2026', u'appearing', u'everywhere', u'\xbf']
     True
     """
     # Replace non-breaking spaces with normal spaces.
     NON_BREAKING = re.compile(u"\u00A0"), " "
     
     # Pad some funky punctuation.
-    FUNKY_PUNCT_1 = re.compile(u'([،;؛¿!"\])}»›”؟%٪°±©®।॥…])'), r" \1 "
+    FUNKY_PUNCT_1 = re.compile(u'([،;؛¿!"\])}»›”؟¡%٪°±©®।॥…])'), r" \1 "
     # Pad more funky punctuation.
     FUNKY_PUNCT_2 = re.compile(u'([({\[“‘„‚«‹「『])'), r" \1 "
     # Pad En dash and em dash


Test patch: