nltk/nltk — Issue #174: WALS interface
Author: alexrudnick
Created: 2012-01-17 06:27:46+00:00
State: open

This module loads WALS database files and provides methods to:
* Search for WALS language codes by language name or ISO code
* View features for a language
* Explore hierarchical (i.e. genealogical) relations of languages

Its intended purpose is as a programmatic interface to the data for use in other tasks.

Data should be downloaded from http://wals.info/export and extracted. The path of the extracted data (likely a directory called "wals_data") is given to the WALS class' __init__ method. By default it expects UTF-8 encoded tab-delimited files, but it also works with comma-separated files (give "dialect='csv'" or "dialect='excel'" as a parameter to the constructor) and latin-1 encodings (give "encoding='latin1'" or "encoding='iso-8859-1'").

In this version, I have removed functionality to output a PDF graph of language genealogical clusters (using graphviz and circo), as the function is not necessary and a bit clunky. It can be re-added if deemed useful.

Also, this version uses hard-coded structures for the Languages, Features, and Values tables, making the assumption that their schema won't change frequently. The Datapoints table is dynamically created based on the features defined, so it should still work if new features are added to WALS. An earlier version dynamically created the other tables based on the header line of each table, but I thought it was unnecessary and lead to less clear code.

Migrated from http://code.google.com/p/nltk/issues/detail?id=708

earlier comments

goodman.m.w said, at 2011-10-30T00:50:10.000Z:

Steven noted that namedtuples are used and thus the module is not compatible with Python 2.5, which is a requirement for the main codebase. nltk_contrib does not need to follow this requirement, however.

If it is preferred, I can replace the namedtuples with something else. Classes would allow the attribute-style accessing of data (e.g. feature.name), but this may be unnecessary. A simple dictionary would be an easy solution, but then data would need to be accessed by a hash key (e.g. feature['name']).

alex.rudnick said, at 2011-10-30T04:10:04.000Z:

Michael, would you do that? That way your module will work on Python 2.5 as well.

goodman.m.w said, at 2011-10-30T22:06:58.000Z:

Certainly. Here is a version without namedtuples. I verified it loads with Python2.5 and tested a few functions. I also added another index and made it a bit more robust to parameters of the wrong type, keyerrors, etc.

alex.rudnick said, at 2011-11-01T15:41:09.000Z:

Michael, this looks good! I checked it in to ntlk_contrib just now.

https://github.com/nltk/nltk_contrib/commit/3e9d1fadb4e4d3e16cfe21406ba621d65da699ff

Would you add a demo() function so that it's clear for potential users how to use it?

goodman.m.w said, at 2011-11-01T20:45:19.000Z:

Thanks Alex!

I'd be happy to add a demo function, but I have a question. The module depends on the WALS database, which AFAIK is not included in nltk_data. It's possible to write Python code to submit an HTTP POST request to http://wals.info/export, then obtain the zip file via a response and decompress it, but there's a couple issues with this:

1. Where would I store the files? A temporary directory?
2. I should first contact the WALS maintainers to see if this sort of automatic downloading is allowed (it bypasses viewing the Terms of Use, and might disrupt how they track usage)

For now, I'm submitting a version with a demo function that requires the data directory as a parameter, and returns with an error message if it's not provided.

Also I noticed a bug in the get_languages_with_feature() function, which is fixed in this version.

alex.rudnick said, at 2011-11-02T05:14:59.000Z:

Hey Michael, that's a good demo! Makes it really clear what the code does.

Pushed at https://github.com/nltk/nltk_contrib/commit/09eeb91b5a25944d34421bce237e7f63f42c0c45 .

Hrm, one downside of Steven removing everybody's commit access to the googlecode site is that I can't close this issue by myself anymore.

goodman.m.w said, at 2011-11-02T07:39:10.000Z:

That's ok Alex. Thanks for all the prompt feedback and help getting the code in!