enthought/distarray — pull request #151 by bgrant (2014-02-25 02:33:31+00:00)
Title: Add HDF5 IO

Don't merge yet, but check this out.

In this PR, on top of #142, I add writing to and reading from HDF5 files.

Leveraging `h5py`'s ability to leverage a parallel build of HDF5, and leveraging our local maps, the `save_hdf5` function on `Context` can tell all the engines to write the indices they own to a *single* dataset in a *single* HDF5 file in parallel! This HDF5 file doesn't have any special knowledge of `DistArray`s; I assume any application that can read HDF5 files can read it.

I also include a simple `load_hdf5` function that uses `fromndarray`. This seems to work, but it should work more predictably and testably after merging in @cowlicks 's fixes for `fromndarray`.

This PR won't pass TravisCI yet; I haven't yet told TravisCI how to build a parallel-enabled HDF5 and `h5py` on top of that, and that could be tricky (as my previous build notes PR suggests).

I've put the `h5py` imports inside the `save...` and `load...` functions in the hopes that `h5py` can be an optional dependency. We'll need to do something about the corresponding tests to make this happen. Perhaps we could have specially named `hdf5test_...`s and a Makefile target for them?