nltk/nltk — Issue #66: Document Iterator for Corpora
Author: alexrudnick
Created: 2012-01-17 06:25:16+00:00
State: open

It would be nice for some applications (LDA, LSA, for instance), if corpus
objects had a document iterator. (Note that iterating over files is not a
substitute, as the BNC and Brown, for instance, put multiple "documents" in
a file.)

As incentive, if this happens, I (jordan boyd-graber) will write an LDA module.

Migrated from http://code.google.com/p/nltk/issues/detail?id=375

earlier comments

StevenBird1 said, at 2011-03-28T00:40:45.000Z:

Sorry for the delay! There's a number of trivial cases, where there is a one-to-one correspondence between documents and files (e.g. Reuters).

The hard cases require detection of document boundaries within files. For the Brown Corpus, it might be possible to use the word range information in the CONTENTS file, e.g.: A01 - A. November 4, 1961, p.1 "Atlanta Primary ..." 0010-0670. However, there's nothing in the files corresponding to the 10-word gap between each document as suggested by these numbers. Perhaps the numbers are relevant to some other version of the corpus. The TEI version of the Brown Corpus also does not indicate the document boundaries. The tagged version of the Brown Corpus included with the Penn Treebank has different formatting, but still does not indicate document boundaries.

I'm not aware of easy cases, where the document boundaries can be detected programmatically. If these existed, then there would be a case for adding new functionality, with new methods (documents(), tagged_documents(), parsed_documents()). If they don't, then we might need to stick with the existing functionality (accessing files, just for those cases where we know there is a one-to-one correspondence with documents), until we figure out how to solve the hard cases such as the Brown Corpus.

jordanbg said, at 2011-03-28T04:15:52.000Z:

There is at least one case where files != documents and can be detected programmatically, which is the BNC. I haven't played around with enough corpora to know other cases.

jordanbg said, at 2011-03-31T19:48:45.000Z:

For future corpora, could we have a corpus-level documents function that by default is not implemented but that downstream classes could implement if there were documents? I would be willing to then do the implementation for existing corpora where documents were relatively straightforward. I just fear that having ad hoc methods (like chapters in Europarl) might cause issues / incompatibilities.