nltk/nltk — Issue #116: Punkt models derived from Wikipedia
Author: alexrudnick
Created: 2012-01-17 06:26:27+00:00
State: open

Currently all Punkt models are trained on news text. Wikipedia or another
large corpus may provide a wider variety of textual information to a
trained model.

One problem with this assertion is that Wikipedia tends to use fewer
abbreviations than print news media; has more stylistic variation (e.g. in
the use of .s for abbreviations and acronyms); and is more likely to
include capitalisation variants that present a problem for the current
binary implementation of Punkt's orthographic context feature.

Ideally, this depends on issues #552 and #553.

Migrated from http://code.google.com/p/nltk/issues/detail?id=554