Project: datamanagement
Issue DM-1887: HDF5 file format study
Xiquin, Loi, Trey, and myself discussed HDF5 as a default format to return result set and metadata from lower-level database services vs. traditional IPAC table. Here is the summary: Advantages of IPAC Table format - Simple and human-readable, contains a single table - Fixed length rows (easy to page through) - Supported by many astronomical tools - Provides a way to pass data type, units, and null values in the header - More metadata can be added through keywords (attributes) Disadvantages of IPAC table format - Steaming can not be started before all data are received – need to know column width before the table can be written (csv is better alternative) - Only alpha-numeric and '_' characters are allowed in column names (small subset of available characters) - Only predefined datatypes and one attribute type (string) - ASCII representation requires about twice as much storage to represent floating-point number data than the binary equivalent. Advantages of HDF5 - Can represent complex data and metadata (according to LOFAR, good to represent time series) - Structured data, arbitrary attribute types, datatypes can be combined to create structured datatypes - Flexible datatypes: can be enumerations, bit strings, pointers, composite datatypes, custom atomic datatypes - Access time and storage space optimizations - Partial I/O: “Chunked” data for faster access - Supports parallel I/O (reading and writing) - Built-in compression (GNU zlib, but can be replaced with others) - Existing inspection and visualization tools (HDFView, MATLAB, etc.) Disadvantages of HDF5 - Complex - Tuned to do efficient I/O and storage for "big" data (hundreds of megabytes and more), not efficient for small reads/writes. - Requires native libraries (available in prepackaged jars, see below) - Not human readable - (?) Not yet widely supported by astronomical tools (counter-examples: AstroPy, IDL, more at hdfgroup site) Tools and Java wrappers: * JHI5 - the low level JNI wrappers: very flexible, but also quite tedious to use. * Java HDF object package - a high-level interface based on JHI5. * HDFView - a Java-based viewer application based on the Java HDF object package. * JHDF5 - a high-level interface building on the JHI5 layer which provides most of the functionality of HDF5 to Java. The API has a shallow learning curve and hides most of the house-keeping work from the developer. You can run the Java HDF object package (and HDFView) on the JHI5 interface that is part of JHDF5, so the two APIs can co-exist within one Java program. (from StackOverflow answer, 2012) * NetCDF-Java is a Pure Java Library, that reads HDF5. However, it's hard to keep pure java version up-to-date with the standard, does not support all the features. A way to set up native libraries (3rd option from JHDF5 FAQ): "Use a library packaged in a jar file and provided as a resource (by putting the jar file on the class path). Internally this uses the same directory structure as method 2., but packaged in a jar file so you don't have to care about it. Jar files with the appropriate structure are cisd-jhdf5-batteries_included.jar and lib/nativejar/.jar (one file for each platform). This is the simplest way to use the library."
Story points: 1