nltk/nltk — pull request #201 by aboSamoor (2012-02-01 23:53:58+00:00)
Title: Complying with the Treebank Tokenization

I compared the results of tokenizing the first 1000 sentence in Penn Treebank using this module with the standard tokenization that I could retrieve from the Parse Trees.

I found that the module actually does not comply with the treebank tokenization for around 150 sentence. With these modification the number of differences drop to around 25 sentences.

I suggest that before merging those changes, we should take advantage of the portion of PTB that NLTK distributes. I do not think the raw text is part of that till now. However, I see it is under fair use to distribute the raw text that is corresponding to the first two sections.

To get the first 1000 sentence from the raw text of PTB, I split the paragraphs manually. I can send you the paragraphs splitted into sentences that cover the first two sections. This is in general will be a good dataset for unit testing for both tasks, sentence splitting and work tokenization. Not to mention possible classifiers that could be trained using such dataset.