nltk/nltk — Issue #115: Make Punkt more robust to large training corpora
Author: alexrudnick
Created: 2012-01-17 06:26:25+00:00
State: open

The Punkt algorithm was designed and tested on 220,000 to 850,000-word
corpora. There may be value in training the system on larger corpora, but
the NLTK implementation is perhaps not robust enough to do so.

In particular, the orthographic context feature is currently implemented
as a bit-mask, meaning that e.g. a single appearance of a word in
lowercase is treated with the same importance as thousands of instances of
the same word capitalised.

ortho_context therefore needs to be re-implemented, in training, as a
counter, with a threshold when training is finalised to determine the
final bit-masks used.

Migrated from http://code.google.com/p/nltk/issues/detail?id=553

earlier comments

peter.ljunglof@heatherleaf.se said, at 2010-07-05T20:29:19.000Z:

I have trained NLTK Punkt on 58 Swedish novels totalling 4.1 M words. Training took about an hour on my 2.5GHz 4GB MacBook, and the resulting pickled model was 3.6 MB.

But it's fast: 100K-200K words/second on average. And loading the pickled model is just a tenth of a second. So I don't complain...

joel.nothman said, at 2010-07-06T03:46:47.000Z:

Speed wasn't my concern here. The algorithm itself is too susceptible to single instances of variation. In particular, unlike other components of their algorithm, orthographic context is entirely binary. If a word *ever* appears un-capitalised in training, that immediately affects its role in sentence boundary ambiguity.

jan.strunk said, at 2010-11-06T22:15:58.000Z:

Hello! Just a quick comment. We actually didn't use bit masks in our original Perl implementation of Punkt, but actually counted all relevant orthographic contexts. So the bit masks must have been introduced in the NLTK implementation.
We did, however, only use binary decision rules in our orthographic heuristic and didn't take the actual counts into account. This was only due to the fact that we didn't put a lot of emphasis on the orthographic heuristic.

I agree that it would be more informative to actually store the raw counts instead of using a bit masks. This would also make it possible to use a somewhat more refined orthographic heuristic (possibly using statistics and/or confidence).
Our orthographic heuristic is also deliberately cautious. It often doesn't take a decision if there is conflicting evidence. (We mainly wanted to implement a robust heuristic that could also cope with upper or lower-case only text.) However, if you are sure that a text corpus is well behaved with regard to capitalization, it could make sense to use a less cautious heuristic that disregards rare cases such as proper names that are always spelt lowercase (such as amnesty international, etc.).

I don't really have time myself at the moment to further develop Punkt, but we currently have three undergrad applied computer science students working on different aspects of sentence boundary detection. One is, for example, looking at the best way to integrate existing lists of abbreviations etc. If their results seem promising, I will probably also propose some improvements for the NTLK version of Punkt.