using nltk tokenizers instead of my own.

replaced home grown regexes with valid tokenizers.