nltk/nltk — pull request #434 by larsmans (2013-07-11 10:25:07+00:00)
Title: improve word_tokenize speed

Runs roughly twice as fast on my test set of 40000 sentences. According to kernprof (http://pythonhosted.org/line_profiler/), the culprits of the previously bad performance were

r"([^' ])('ll|'re|'ve|n't|) "

and

r"([^' ])('LL|'RE|'VE|N'T|) "

which were matching almost all words due to the | at the end.

Also removed some useless whitespace handling that was being undone by the `str.split` just before return.

The contractions handling is now the slowest part @~20% of the time.

Tests pass.