nltk/nltk — Issue #124: A more robust Penn Treebank corpus reader
Author: alexrudnick
Created: 2012-01-17 06:26:37+00:00
State: open

I have customized my NLTK installation with a new, more robust corpus reader for the full Penn Treebank (both WSJ and Brown portions). This corpus reader recognizes the categorization of portions of the Treebank into genres (news = WSJ, mystery, romance, etc.):

>>> from nltk.corpus import ptb
>>> ptb.categories()
['adventure', 'belles_lettres', 'fiction', 'humor', 'lore', 'mystery', 'news', 'romance', 'science_fiction']
>>> ptb.parsed_sents(categories='science_fiction')[1]
Tree('S', [Tree('NP-SBJ', [Tree('NP', [Tree('NN', ['Self']), Tree('POS', ["'s"])]), Tree('NN', ['integrity'])]), Tree('VP', [Tree('VP', [Tree('VBD', ['was'])]), Tree('CC', ['and']), Tree('VP', [Tree('VBZ', ['is'])]), Tree('CC', ['and']), Tree('VP', [Tree('ADVP-TMP', [Tree('RB', ['ever'])]), Tree('VBD', ['had']), Tree('VP', [Tree('VBN', ['been'])])])]), Tree('.', ['.'])])

Specifically, I have created the class CategorizedBracketParseCorpusReader—a subclass of both CategorizedCorpusReader and BracketParseCorpusReader—because WSJ and Brown articles contain parses and are also grouped into topical categories.

Since this seems like it would be useful to others working with the Treebank, I am posting the changes here; see the attached patch (relative to 2.0b9).

Migrated from http://code.google.com/p/nltk/issues/detail?id=589

earlier comments

StevenBird1 said, at 2010-08-27T05:48:40.000Z:

Thanks Nathan. Does the user of this corpus reader need to work out what to put in the file allcats.txt? How would we select just the WSJ section of Penn Treebank (since this is not a "category")?

neatnate said, at 2010-08-27T13:15:11.000Z:

Here's my allcats.txt. My understanding is that the WSJ section is the entire 'news' category.