timholy/HDF5.jl — pull request #132 by simonster (2014-08-13 18:58:07+00:00)
Title: Write Julia types as HDF5 compound types

This is a major change to the serialization format of JLD. Rather than writing Julia types as arrays of references, we translate Julia types to HDF5 compound types and write those. Following the discussion in #27, everything Julia would store inline is presently stored inline in the compound type, and everything Julia would store as a pointer to another object is stored as an HDF5 reference to another object. This PR also incorporates the changes from #102.

This presently passes all read and write tests, but there is plenty more work to be done:

- We cannot yet read JLD files created with earlier versions of JLD or convert them to the new format.
- There is no validation to ensure that HDF5 types match Julia types. Thus, if a Julia type changes, JLD will probably segfault on readout.
- There is no ability to read data if the type is not available in the current workspace. The structure of the compound type is sufficient to reconstruct the Julia type hierarchy (except for the types of reference fields) but this is not yet implemented.
- The `rootmodule` keyword argument to `write` is not yet supported. This is a bit painful to deal with if the user decides to write the same type with and without the `rootmodule` keyword, since we would need to create two distinct compound types. @timholy, can you explain the use case for this so I can think about whether there's an easier design that could accomplish the same ends?
- This code has a lot more cases to handle than the old code and it is unlikely that the existing tests cover all of them.
- In principle, for pointerfree Julia types, HDF5 compound types could be generated with the same padding so that they exactly match the Julia type. This would avoid a copy when reading/writing and also allow mmapping arrays directly from the file, at the cost of saving the padding to the file.
- In some cases where Julia stores fields as pointers to objects, we may want to store the contents inline in JLD. Two such cases are fields that are non-pointerfree immutables and fields that are leaf type tuples. The main complication to storing such fields inline in JLD is that they could be undefined.
- I have to fix some things for Julia 0.2 if we want to maintain compatibility.

Implementation notes:

- Compound types are committed with numeric indices in the `_types` group, with a `julia type` attribute that indicates the corresponding Julia type. Because libhdf5 loses track of compound type hierarchy between closing and reopening a file, numeric suffixes indicate the corresponding compound type index for a field that is itself a compound type.
- Code to convert between a given Julia type and the corresponding HDF5 compound type and vice versa is dynamically generated. This should be very efficient after warmup. The code that does this needs to be inspected very carefully, since it does some somewhat unclean things, especially for non-pointerfree immutables.
- Empty types/immutables and arbitrary bitstypes are stored as HDF5 opaque types. A special case throws an error when a pointer is encountered. As a result, most write methods now throw an error when they encounter a pointer rather than displaying a warning. The variant of `@save` that saves all top-level variables catches the error and displays a warning instead.