nltk/nltk — Issue #59: third-party download locations
Author: alexrudnick
Created: 2012-01-17 06:25:06+00:00
State: open

nltk.download and package indexing should support third-party download
locations, including nltk.ldc.upenn.edu.

This needs to be mentioned in the book as a place where people can upload
their own datasets.

Is this something we can commit to?

Migrated from http://code.google.com/p/nltk/issues/detail?id=318

earlier comments

StevenBird1 said, at 2009-05-03T10:16:27.000Z:

This is mentioned in the book, so we need to do it.

StevenBird1 said, at 2009-07-05T22:39:36.000Z:

This was deleted from the book at the last minute. However, this is the only
scalable way to distribute corpora.

Note that the recent addition of NomBank 1.0 copied the existing distribution into
the NLTK repository unchanged. It might have been better just to point to an
externally hosted distribution. The XML files defining corpus collections could
contain URLs in addition to local identifiers. E.g.:

would become:

The downloader would need to skip over any corpora that couldn't be accessed for
whatever reason.

nmadnani said, at 2009-07-16T01:36:11.000Z:

I would be happy to take a stab at resolving this issue if someone isn't already.

StevenBird1 said, at 2009-07-16T01:48:09.000Z:

We should also support externally hosted collections, e.g.:
http://nltk.ldc.upenn.edu/index.xml

StevenBird1 said, at 2009-07-16T01:57:53.000Z:

Please go ahead. I think tools/build_pkg_index.py needs to be modified to handle the
case described in comment 2, and nltk.downloader.Downloader._url needs to be modified
to support a list of URLs (no longer hard-coded, but read from a standard location,
e.g. http://nltk.googlecode.com/svn/trunk/nltk_data/sites.xml)

nmadnani said, at 2009-07-16T02:51:07.000Z:

It's not clear to me exactly what functionality to add in order to support third-party downloads.

For example, How will build_index() from downloader.py (which is what's used in build_pkg_index.py) know
whether or not to add a url in the xml file? Will the XML file describing the corpus (abc.xml, indian.xml etc.)
contain this information?

How will the URLs from a standard location (e.g., the sites.xml URL you mention above) fit into the picture?

I think a brief description of how the current downloading process works and what it lacks will be very useful
to me to figure out what to do next.

StevenBird1 said, at 2009-07-16T03:32:19.000Z:

Let me expand the proposal a bit. Comment 2 is a proposal to extend the scope of the
corpus XML metadata files (like abc.xml) to contain full URLs. Build_pkg_index
searches for such XML files when it creates its index.xml file. If we only did this,
and put the europarl_raw.xml file
(http://nltk.ldc.upenn.edu/packages/corpora/europarl_raw.xml) into the repository,
we'd be done.

A separate issue is to support the hosting of external *collections* (Comment 4) that
could in principle be maintained by others, plus a file containing the URLs for all
those collections. Let's not bother with that just yet.

nmadnani said, at 2009-07-16T03:46:07.000Z:

Okay, that makes sense to me. Here's the current abc.xml

What would the new version, one that contained a URL, look like? Something like this may be?

So, all I have to do in build_index() is to determine whether an XML file for a package contains such a 'ref'
attribute and if so, just use that as the "url" field instead of using the provided base-url to create the value for
the "url" field.

Does that make sense?

nmadnani said, at 2009-07-16T04:58:55.000Z:

OK, I made the trivial change in build_index() as follows:

for pkg_xml, zf, subdir in _find_packages(os.path.join(root, 'packages')):
zipstat = os.stat(zf.filename)

# Check if the XML file contains a 'ref' attribute. If it does,
# then use that value as the URL instead of constructing one
url = pkg_xml.get('ref', None)
if not url:
url = '%s/%s/%s' % (base_url, subdir, os.path.split(zf.filename)[1])

I tested it and it works as intended. Can I go ahead and commit?

StevenBird1 said, at 2009-07-16T06:13:12.000Z:

Sure. Please also add the europarl_raw.xml file I mentioned (edit as needed), and
rebuild the data index using "make pkg_index" (in trunk/nltk).

StevenBird1 said, at 2009-07-16T06:14:15.000Z:

Note that there's going to be a problem with calculating checksums for remote
corpora, which means we might need to investigate my suggestion for external collections.

nmadnani said, at 2009-07-16T13:41:20.000Z:

Yes, that's a good point. In general, build_index() can't compute any of the statistics (size, unzipped size,
checksum) etc. if we don't have the actual zip file.

I don't see how using external collections would help with this though. I must be missing something ...

StevenBird1 said, at 2009-12-03T01:14:13.000Z:

I think the only easy solution is to manually provide checksums in the XML file. The
XML file should live with the data, which means that the index building process needs
to know some external locations where NLTK corpora live, i.e. URLs of files that
contain lists of NLTK corpora.