nltk/nltk — Issue #78: Tokenizer interface should give access to token locations
Author: alexrudnick
Created: 2012-01-17 06:25:34+00:00
State: open

Add a method to all tokenizers to produce a list of start locations (or
spans) for each token in the input string, cf
http://groups.google.com/group/nltk-users/browse_thread/thread/5423aecbdd8df8db?hl=en

Migrated from http://code.google.com/p/nltk/issues/detail?id=438