nltk/nltk — Issue #181: Include Dan Gillick's 'splitta' Sentence Segmenter
Author: alexrudnick
Created: 2012-01-17 06:27:56+00:00
State: open

Dan Gillick has written a supervised sentence segmenter and it is claimed to have very good performance on well-formed text. Since it's written in python, it should be easy to include that into NLTK in a future release. It has an error rate much, much smaller than what the Punkt model gets. He shows that in his paper and in the README on splitta website:
http://splitta.googlecode.com/

Migrated from http://code.google.com/p/nltk/issues/detail?id=719

earlier comments

joel.nothman said, at 2011-09-20T05:02:29.000Z:

One of the advantages of Punkt is that it is language and domain-adaptable, without gold-standard training. Though I admit that many of Punkt's errors seem avoidable, the question here is what the purpose is in including multiple SBD tools. It may also be worthwhile evaluating splitta on other domains.

It might be interesting to include a serialised splitta model in addition to punkt for comparison.

nmadnani said, at 2011-09-20T16:12:56.000Z:

That might be a theoretical advantage of Punkt but on all the noisy domains I have tried it on, it has failed pretty badly. However, I agree with you that we should evaluate splitta on other domains and, in fact, I am planning to do so on the essay data that we have here at ETS; it is pretty darned noisy and one of the genres that Punkt does poorly on.

As for including multiple SBDs, I am not sure I see an issue. We have different word tokenizers and multiple PoS taggers. We can keep Punkt as the default but why not include other choices for folks who want it for domains on which Punkt does not perform well?

joel.nothman said, at 2011-09-20T23:36:52.000Z:

I wasn't suggesting that having multiple tools was an issue. And it might be worthwhile to use splitta in SBD on NLTK corpora. In short, there's no problem including it from my perspective. But I don't know how it fits into NLTK design, releases, bloat issues... This also assumes that Dan wants it included.

nmadnani said, at 2011-09-20T23:40:53.000Z:

Understood. Splitta is already released under the Apache License so that's one less obstacle. I will contact Dan to see what he thinks.

StevenBird1 said, at 2011-09-20T23:56:27.000Z:

It would be a welcome addition. Note that in order to fit into NLTK, the code should be changed to use new-style classes, have epydoc docstrings, and support the existing tokenizer interface.

nmadnani said, at 2011-09-21T00:20:44.000Z:

So, I contacted Dan and he is totally okay with it. I am going to see if this performs well on my dataset and if it does, I will be motivated to integrate the code myself relatively soon. If it doesn't, it's going to have to move down the queue :)

StevenBird1 said, at 2011-09-21T06:50:52.000Z:

Sounds good. If you modify it to incorporate it in NLTK, I wonder if Dan would like to continue to maintain it -- i.e. merge his googlecode project into NLTK.