nltk/nltk — Issue #135: Sentence tokenizer for TaggedCorpusReader has confusing behavior
Author: alexrudnick
Created: 2012-01-17 06:26:52+00:00
State: open

When using RegxpTokenizer to tokenize tagged text into list of sentence with pattern "[,.] " discard_empty setting to True, the returned result of reader.sents()
still contain empty string.
This is because the tokenizer remove "," and "." from text but still keep the tag for "," and ".". Consequently after reader split word and tag, the empty string before tag will still
be returned.
Personally I find this behavior disturbing.
But is it a bug or a feature? :-)

Migrated from http://code.google.com/p/nltk/issues/detail?id=618

earlier comments

mxf3306 said, at 2011-01-07T14:15:08.000Z:

P.S. I realize that to change this behavior may require API to change.
But since splitting sentences by punctuation would be a common task
and it would be intuitive to just use "," instead of ",/wd" to split,
I think it might be better to provide some additional routine to accomplish this task.