Repository: nltk/nltk

Issue:
TweetTokenizer hangs
I have the following code that I use to  TweetTokenize a day worth of tweet collection, ~2.2M tweets.

```
import pandas as pd
from nltk.tokenize import TweetTokenizer

tknz = TweetTokenizer()

df_25 = pd.read_csv('2015-09-25.csv', header=None,\
                     names=['ID', 'ORIG', 'LAT', 'LONG', 'CITY', 'STATE', 'TIMESTAMP', 'TWEET_ID', 'TWEET'], \
                     low_memory=False, \
                     parse_dates=['TIMESTAMP'], usecols=['TIMESTAMP', 'TWEET'])

twt25=[]
for w in df_25['TWEET']:
    twt25.append(tknz.tokenize(w.lower()))
    #debugging output
    print(w)
    print(len(twt25))
```

The code works as intended unless I come across certain types of tweets, I confirmed this through couple of tests.

For that specific file it was hanging for a long period and couldn't figure out why. I added the `print()` function to find out where the issue lies. Last time I ran it before manually interrupting the script was 30+minutes., so I decided to remove it with Sublime Text and reran it.

The tweet being: `@bunicole http://t.co/LZbFWsdMnl\n!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ITS YOU!` I'm assuming it's because there's 57 `!`

After removal it stopped again on `http://t.co/8K2bEJe9Hn\n???????????????????????? http://t.co/PhylB6ufxw`. However, that one having 24 `?` was for a shorter period of time ~10s. 

Is it possible to skip those types of lines without having to manually delete them?



Discussion:
I think we should try fixing the pathological regex; it looks like a security issue.

I created a csv file with several tweets, adding your entry at the end (I added an ID column just for test):

> 624492026386624513,@bunicole http://t.co/LZbFWsdMnl\n!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ITS YOU!

Then I parsed it using pandas (following your code):

```
import pandas as pd
from nltk.tokenize import TweetTokenizer

tknz = TweetTokenizer()

df_25 = pd.read_csv('test_file.csv', header=None,\
                     names=['TWEET_ID', 'TWEET'], \
                     low_memory=False, \
                     usecols=['TWEET_ID', 'TWEET'])

twt25=[]
for w in df_25['TWEET']:
    twt25.append(tknz.tokenize(w.lower()))
    #debugging output
    print(w)
    print(len(twt25))
```

I found out that the entry is sligthly different after pandas parsing:

```
>>> df_25['TWEET'][5001]
'@bunicole http://t.co/LZbFWsdMnl\\n!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ITS YOU!'
```

You can see that the `\n` has been modified, and it is now escaped as `\\n` (see [here](https://groups.google.com/forum/#!topic/pydata/-kCo7OYAJaE) for another example). I think that this is because pandas uses `\n` to create new rows (but I did not use pandas so much, so I might be wrong). It seems that this is what causes the regexp to take so much time. In fact:

```
>>> from nltk.tokenize.casual import *
>>> tweet_1 = "@bunicole http://t.co/LZbFWsdMnl\n!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ITS YOU!"
>>> tweet_2 = "@bunicole http://t.co/LZbFWsdMnl\\n!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ITS YOU!"
>>> tknz.tokenize(tweet_1)
['@bunicole', 'http://t.co/LZbFWsdMnl', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', '!', 'ITS', 'YOU', '!']
>>> tknz.tokenize(tweet_2)
Takes a lot of time.
```

The regexp that gets stuck is `WORD_RE` in [`tokenize/casual.py`](https://github.com/nltk/nltk/blob/07bcb7ed51260f5da07be841aebf23235b6af96e/nltk/tokenize/casual.py). 


Resolving patch:
diff --git a/nltk/tokenize/casual.py b/nltk/tokenize/casual.py
--- a/nltk/tokenize/casual.py
+++ b/nltk/tokenize/casual.py
@@ -176,6 +176,9 @@
 WORD_RE = re.compile(r"""(%s)""" % "|".join(REGEXPS), re.VERBOSE | re.I
                      | re.UNICODE)
 
+# WORD_RE performs poorly on these patterns:
+HANG_RE = re.compile(r'([^a-zA-Z0-9])\1{3,}')
+
 # The emoticon string gets its own regex so that we can preserve case for
 # them as needed:
 EMOTICON_RE = re.compile(EMOTICONS, re.VERBOSE | re.I | re.UNICODE)
@@ -295,8 +298,10 @@ def tokenize(self, text):
         # Normalize word lengthening
         if self.reduce_len:
             text = reduce_lengthening(text)
+        # Shorten problematic sequences of characters
+        safe_text = HANG_RE.sub(r'\1\1\1', text)
         # Tokenize:
-        words = WORD_RE.findall(text)
+        words = WORD_RE.findall(safe_text)
         # Possibly alter the case, but avoid changing emoticons like :D into :d:
         if not self.preserve_case:
             words = list(map((lambda x : x if EMOTICON_RE.search(x) else


Test patch:
diff --git a/nltk/test/tokenize.doctest b/nltk/test/tokenize.doctest
--- a/nltk/test/tokenize.doctest
+++ b/nltk/test/tokenize.doctest
@@ -157,3 +157,10 @@ The `preserve_case` parameter (default: True) allows to convert uppercase tokens
     >>> s9 = "@jrmy: I'm REALLY HAPPYYY about that! NICEEEE :D :P"
     >>> tknzr.tokenize(s9)
     ['@jrmy', ':', "i'm", 'really', 'happyyy', 'about', 'that', '!', 'niceeee', ':D', ':P']
+
+It should not hang on long sequences of the same punctuation character.
+
+    >>> tknzr = TweetTokenizer()
+    >>> s10 = "Photo: Aujourd'hui sur http://t.co/0gebOFDUzn Projet... http://t.co/bKfIUbydz2.............................. http://fb.me/3b6uXpz0L"
+    >>> tknzr.tokenize(s10)
+    [u'Photo', u':', u"Aujourd'hui", u'sur', u'http://t.co/0gebOFDUzn', u'Projet', u'...', u'http://t.co/bKfIUbydz2', u'...', u'http://fb.me/3b6uXpz0L']
\ No newline at end of file