timholy/HDF5.jl — pull request #22 by simonster (2013-06-24 22:16:22+00:00)
Title: Add option to mmap arrays from JLD files

It turns out it's relatively easy to get `libhdf5` to provide the offset of a dataset within a given HDF5 file, which gives us all we need to be able to be able to mmap provided that they are uncompressed and contiguously stored. This approach has two major advantages over the hyperslab interface:

1. There is no overhead involved in reading/writing individual elements or slices from an array beyond that of disk IO. Even the IO is faster, since it isn't necessary to copy from kernel to user space.
2. Since `mmap_array` returns a regular Julia arrays, they can be incorporated into Julia objects, which means that any Julia type that stores its underlying data in arrays can be effectively mmapped.

This pull request adds a `mmaparrays` named parameter to `jldopen()`, which loads arrays of `HDF5BitsKind` using `mmap` instead of the relevant HDF5 library methods. This seemed too easy, so it's possible there are cases I haven't thought about.

See also HarlanH/DataFrames.jl#25 (although it looks like the `jld_dataframe` tests are currently failing due to `#undef`s).