nltk/nltk — pull request #746 by smithsimonj (2014-09-22 08:56:00+00:00)
Title: Added comma condition to PunktWordTokeniser

This addition to the word tokeniser splits words that are separated by commas but not numbers. For example the sentence:
'I have 13,000,000 bottles,in,my,bag'
tokenises as:
['I', 'have', '13,000,000', 'bottles', ',', 'in', ',', 'my', ',', 'bag']
rather than the awkward:
['I', 'have', '13,000,000', 'bottles,in,my,bag']

This is useful as this occurs quite a lot in blogs and tweets and has caused errors in my processing chain when we had some 1600 words joined in this way - leading to a massive word being created...