ContinuumIO/blaze — pull request #818 by mrocklin (2014-11-10 02:08:47+00:00)
Title: Improve discovery and navigation of h5py files

Turns out that navigating complex h5py files can be tricky. This extends `discover` and `compute_up(Field, ...)` to h5py.File and h5py.Group. This provides easier inspection

Here is an example use

```Python
In [1]: import h5py

In [2]: f = h5py.File('/home/mrocklin/Downloads/OMI-Aura_L2-OMAERO_2014m1105t2304-o54838_v003-2014m1106t215558.he5')

In [3]: f.keys() # One way to navigate h5py files normally
Out[3]: [u'HDFEOS', u'HDFEOS INFORMATION']

In [4]: f['HDFEOS'].keys()
Out[4]: [u'ADDITIONAL', u'SWATHS']

In [5]: from blaze import Data, into, np

In [6]: d = Data(f)

In [7]: d.
d.HDFEOS d.dshape d.isidentical d.map d.schema
d.data d.fields d.like d.relabel

In [7]: d.HDFEOS.
d.HDFEOS.ADDITIONAL d.HDFEOS.fields d.HDFEOS.relabel
d.HDFEOS.SWATHS d.HDFEOS.isidentical d.HDFEOS.schema
d.HDFEOS.dshape d.HDFEOS.map

In [7]: d.HDFEOS.SWATHS.
d.HDFEOS.SWATHS.ColumnAmountAerosol d.HDFEOS.SWATHS.map
d.HDFEOS.SWATHS.dshape d.HDFEOS.SWATHS.relabel
d.HDFEOS.SWATHS.fields d.HDFEOS.SWATHS.schema
d.HDFEOS.SWATHS.isidentical

In [7]: d.HDFEOS.SWATHS.ColumnAmountAerosol.
d.HDFEOS.SWATHS.ColumnAmountAerosol.dshape
d.HDFEOS.SWATHS.ColumnAmountAerosol.fields
d.HDFEOS.SWATHS.ColumnAmountAerosol.isidentical
d.HDFEOS.SWATHS.ColumnAmountAerosol.map
d.HDFEOS.SWATHS.ColumnAmountAerosol.relabel
d.HDFEOS.SWATHS.ColumnAmountAerosol.schema

In [7]: d.HDFEOS.SWATHS.ColumnAmountAerosol.fields
Out[7]: [u'Data Fields', u'Geolocation Fields']

In [8]: d.HDFEOS.SWATHS.ColumnAmountAerosol['Data Fields'] # TODO: need to improve tab completion on fields with spaces
Out[8]:

In [9]: g = d.HDFEOS.SWATHS.ColumnAmountAerosol['Data Fields'] # TODO: need to improve tab completion on fields with spaces
In [10]: g.
g.AerosolIndexUV
g.AerosolIndexVIS
g.AerosolModelMW
g.AerosolModelsPassedThreshold
g.AerosolOpticalThicknessMW
g.AerosolOpticalThicknessMWPrecision
g.AerosolOpticalThicknessNUV
g.AerosolOpticalThicknessPassedThreshold
g.AerosolOpticalThicknessPassedThresholdMean
g.AerosolOpticalThicknessPassedThresholdStd
g.CloudFlags
g.CloudPressure
g.EffectiveCloudFraction
g.InstrumentConfigurationId
g.MeasurementQualityFlags
g.NumberOfModelsPassedThreshold
g.ProcessingQualityFlagsMW
g.ProcessingQualityFlagsNUV
g.RootMeanSquareErrorOfFitPassedThreshold
g.SingleScatteringAlbedoMW
g.SingleScatteringAlbedoMWPrecision
g.SingleScatteringAlbedoNUV
g.SingleScatteringAlbedoPassedThreshold
g.SingleScatteringAlbedoPassedThresholdMean
g.SingleScatteringAlbedoPassedThresholdStd
g.SmallPixelRadiancePointerUV
g.SmallPixelRadiancePointerVIS
g.SmallPixelRadianceUV
g.SmallPixelRadianceVIS
g.SmallPixelWavelengthUV
g.SmallPixelWavelengthVIS
g.TerrainPressure
g.TerrainReflectivity
g.XTrackQualityFlags
g.dshape
g.fields
g.isidentical
g.map
g.relabel
g.schema

In [12]: into(np.ndarray, g.SingleScatteringAlbedoMW)
```

cc @jdmcbr

Also, `discover` works.

```Python
In [20]: discover(f)
Out[20]: dshape("{ HDFEOS : { ADDITIONAL : { 'FILE_ATTRIBUTES' : { } }, SWATHS : { ColumnAmountAerosol : { 'Data Fields' : { AerosolIndexUV : 1643 * 60 * int16, AerosolIndexVIS : 1643 * 60 * int16, AerosolModelMW : 1643 * 60 * uint16, AerosolModelsPassedThreshold : 1643 * 60 * 10 * uint16, AerosolOpticalThicknessMW : 1643 * 60 * 14 * int16, AerosolOpticalThicknessMWPrecision : 1643 * 60 * int16, AerosolOpticalThicknessNUV : 1643 * 60 * 2 * int16, AerosolOpticalThicknessPassedThreshold : 1643 * 60 * 10 * 9 * int16, AerosolOpticalThicknessPassedThresholdMean : 1643 * 60 * 9 * int16, AerosolOpticalThicknessPassedThresholdStd : 1643 * 60 * 9 * int16, CloudFlags : 1643 * 60 * uint8, CloudPressure : 1643 * 60 * int16, EffectiveCloudFraction : 1643 * 60 * int8, InstrumentConfigurationId : 1643 * uint8, MeasurementQualityFlags : 1643 * uint8, NumberOfModelsPassedThreshold : 1643 * 60 * uint8, ProcessingQualityFlagsMW : 1643 * 60 * uint16, ProcessingQualityFlagsNUV : 1643 * 60 * uint16, RootMeanSquareErrorOfFitPassedThreshold : 1643 * 60 * 10 * int16, SingleScatteringAlbedoMW : 1643 * 60 * 14 * int16, SingleScatteringAlbedoMWPrecision : 1643 * 60 * int16, SingleScatteringAlbedoNUV : 1643 * 60 * 2 * int16, SingleScatteringAlbedoPassedThreshold : 1643 * 60 * 10 * 9 * int16, SingleScatteringAlbedoPassedThresholdMean : 1643 * 60 * 9 * int16, SingleScatteringAlbedoPassedThresholdStd : 1643 * 60 * 9 * int16, SmallPixelRadiancePointerUV : 1643 * 2 * int16, SmallPixelRadiancePointerVIS : 1643 * 2 * int16, SmallPixelRadianceUV : 6783 * 60 * float32, SmallPixelRadianceVIS : 6786 * 60 * float32, SmallPixelWavelengthUV : 6783 * 60 * uint16, SmallPixelWavelengthVIS : 6786 * 60 * uint16, TerrainPressure : 1643 * 60 * int16, TerrainReflectivity : 1643 * 60 * 9 * int16, XTrackQualityFlags : 1643 * 60 * uint8 }, 'Geolocation Fields' : { GroundPixelQualityFlags : 1643 * 60 * uint16, Latitude : 1643 * 60 * float32, Longitude : 1643 * 60 * float32, OrbitPhase : 1643 * float32, SolarAzimuthAngle : 1643 * 60 * float32, SolarZenithAngle : 1643 * 60 * float32, SpacecraftAltitude : 1643 * float32, SpacecraftLatitude : 1643 * float32, SpacecraftLongitude : 1643 * float32, TerrainHeight : 1643 * 60 * int16, Time : 1643 * float64, ViewingAzimuthAngle : 1643 * 60 * float32, ViewingZenithAngle : 1643 * 60 * float32 } } } }, 'HDFEOS INFORMATION' : { 'ArchiveMetadata.0' : string[65535, 'A'], 'CoreMetadata.0' : string[65535, 'A'], 'StructMetadata.0' : string[32000, 'A'] } }")
```

We should improve pretty printing of datashapes. This could be much more helpful.