nltk/nltk — Issue #114: Pickled Punkt models need language specific variables
Author: alexrudnick
Created: 2012-01-17 06:26:24+00:00
State: open

Issue #479 highlights the need for language-specific variables in our
Punkt implementation (e.g. Greek's use of ';' in place of English '?').

Currently the Punkt models in nltk_data/tokenizers/punkt.zip are derived
using the same default (English-based) variables.

The punkt module needs to incorporate non-default variables as necessary,
and the pickles need to be reproduced.

Migrated from http://code.google.com/p/nltk/issues/detail?id=552

earlier comments

jan.strunk said, at 2010-05-11T14:40:42.000Z:

That's certainly right. Since we only concentrated on the more ambiguous period . in
our original Punkt research and prototype implementation and not on additional
punctuation marks, I do not have any ready-made punctuation sets for the different
languages. I would, however, be happy to reproduce the pickles if knowledgeable
people would provide such localized versions for different languages. I could also
provide such localized variables for German.