nltk/nltk — pull request #487 by deanmalmgren (2013-09-18 15:43:03+00:00)
Title: word_tokenizer / TreebankWordTokenizer rather slow

I'm tokenizing several million documents and need to tokenize each of them separately. When tokenizing each of these documents, the repeated calls to re.sub recompiles each of [all these regular expressions on each document](https://github.com/nltk/nltk/blob/master/nltk/tokenize/treebank.py#L61-L87) and is consequently rather slow. Is there a reason that each of these regular expressions aren't pre-compiled as class variables like the [contractions regular expressions](https://github.com/nltk/nltk/blob/master/nltk/tokenize/treebank.py#L46-L57)?

My understanding is that [pre-compiling is 2-3x faster](http://stackoverflow.com/a/452142/564709).