nltk/nltk — Issue #131: [patch] implement span_tokenize() for Treebank word tokenizer
Author: alexrudnick
Created: 2012-01-17 06:26:47+00:00
State: open

The Treebank word tokenizer doesn't implement span_tokenize() yet. This patch adds it and re-implements tokenize() in terms of it. Full disclosure: I did some limited testing to see if performance was affected, and it seems the new method is maybe 25% slower.

Migrated from http://code.google.com/p/nltk/issues/detail?id=614