colah/data — pull request #6 by colah (2013-11-18 01:45:03+00:00)
Title: New dataset model: handle.py + arbitrary files.

We move to a new model of datasets, where the only requirement is
that a dataset provide a `handle.py` file describing how to use it.

This model, along similar ideas as Kolmogorov Complexity, allows
us to support the great breadth of data used in the sciences. It
simply isn't practical to coerce all data into a single format.

To avoid a hundred implementations of importers for common formats,
we introduce "helpers". These are available to `handle.py` at load
time, and can handle importing common data types.

This commit has had some basic testing. Datasets in datasets.py are
not moved to the new model at the time of this commit. There is
basic documentation of the new model, and a template directory with
a template `handle.py`.