Package: simple-h5py
simple h5pyA simple wrapper for the h5py libraryimportnumpyasnpfromsimple_h5pyimportBasicH5File# Creating some data# >> notice the "huge" attribute !group_attrs=dict(a=1,b=2)dataset=np.ones((5,4,3))dataset_attrs=dict(new=5,huge=np.ones((1000000,3)))# Write contents to fileobj=BasicH5File('demo.h5')obj['my_group']=Noneobj['my_group'].attrs=group_attrs.copy()obj['my_group']['my_dataset']=datasetobj['my_group']['my_dataset'].attrs=dataset_attrs.copy()# Read contents from fileobj=BasicH5File('demo.h5')print(obj['my_group'].attrs)print(obj['my_group']['my_dataset'][0])print(obj['my_group']['my_dataset'].attrs)The above snippet creates a HDF5 file with the following "content tree".demo.h5
│──my_group
│├──.attrs
││├──a(1)││└──b(2)│└──my_dataset(array)│└──.attrs
│├──new(5)│└──huge(arrayref)└──big_attrs└──my_group.my_dataset.attrs.huge(array)Seebelowfor the equivalent snippet usingvanillah5py.ContentsInstallation instructionsLibrary FeaturesFull Python experienceIntelligent open/close of fileHandling of BIG ATTRIBUTESNiceprintof the content-treeDefine required attributesAdditional notesEquivalent snippet usingh5pyInstallation instructionsFrom PyPIpipinstallsimple_h5pyFrom Condacondainstall-carturo.mendoza.quispesimple_h5pyFrom gitpipinstallgit+https://github.com/amq92/simple_h5py.gitLibrary Features:rocket: Full Python experienceWhileh5pydoes provide ahigh-level interface to the HDF5 library using established Python and NumPy concepts, it purposely does not go the extra mile to provide a full Python experience to the user.
Of course, this design choice allows for great flexibility such as enabling chunked storage, allowing to store and manipulate the data in memory (using thecoredriver) and much more !The goal ofsimple_h5pyis to allow easier creation and handling of HDF5 files using the fabuloush5pylibrary as a support :+1:The following example highlights the similarity between creating a python-only object and creating a HDF5 file usingsimple_h5py!obj=dict()obj['group']=dict()obj['group']['attrs']=dict(a=1,b=2)obj['group']['dataset']=dict(contents=dataset)obj['group']['dataset']['attrs']=dict(c=3,d=4)obj=BasicH5File('myfile.h5')obj['group']=Noneobj['group'].attrs=dict(a=1,b=2)obj['group']['dataset']=datasetobj['group']['dataset'].attrs=dict(c=3,d=4):zap: Intelligent open/close of fileIf the library is used forcreatinga new HDF5 file, it will open the file stream whenever a new value is given (either to a group, a dataset or an attribute instance) and subsequently closes it.
This is done only once for each__setitem__call.
Hence, setting an entire attribute group with a single dictionary will only require one open/close directive.
However, setting each attribute group entry will require multiple ones.It the library is used forreadinga new HDF5 file, it will parse the complete content-tree and load it into memory.
Additionally, it will load all group and dataset attributes too.
However, thedatasets will not be loadedsince they are assumed to be heavy.
Since the returned object contains the correct HDF5 references, the datasets can be loaded at any time (either completely or partially).obj=BasicH5File('myfile.h5')# Load content-tree & attrs from diskv1=obj['group'].attrs# Inspect object in memoryv2=obj['group'].attrs['a']# Inspect object in memoryv3=obj['group']['dataset'].attrs# Inspect object in memoryv4=obj['group']['dataset'].attrs['c']# Inspect object in memoryv5=obj['group']['dataset'][:]# Load complete dataset from diskv6=obj['group']['dataset'][:10]# Partially load dataset from diskThis strategy should allow a morefluidinteraction with the HDF5 file since it can be fully inspected at any time, without requiring multiple open/close directives !:earth_americas: Handling of BIG ATTRIBUTESThe section of HDF5 User's Guide dedicated to the case ofLarge Attributes.
Since,Attributes are intended to be small objects, most implementations limit the size of these meta-data (h5pywill throw a RuntimeError).
The User's Guide proposes to point the attribute to another supplemental dataset.simple_h5pyimplements this and makes the issue completely transparent to the user.
Every large attribute will be stored into a dataset with full path/big_attrs/<dataset_name>.<group_name>.attrs.<attribute_name>.obj=BasicH5File('myfile.h5')dst=obj['group']['dataset']dst.attrs['e']=np.ones((1000000,3))# silently creates large dataset in# '/big_attrs/group.dataset.attrs.e'dst.attrs['f']=np.ones((10,3))# normal attribute creation but with# identical syntax !Note thatsimple_h5pyautomatically handles thehugeattribute during both, reading and writing the contents !obj=BasicH5File('myfile.h5')v7=obj['group']['dataset'].attrs['e']# they are identical for the user !v8=obj['group']['dataset'].attrs['f']:pencil2: Niceprintof the content-treeSince theBasicH5Filecontains the entire content-tree at all times, displaying the object (either__repr__or__str__) allows for fast inspection of the file contents.print(obj)# BasicH5File (myfile.h5)# > Group "/group"#   > Dataset "/group/dataset" (20, 30, 10)display(obj['group'])# Group# > path: myfile.h5# > route: /group# > attrs: {'a': 1, 'b': 2}# > datasets: ['dataset']obj['group']['dataset']# Dataset# > path: myfile.h5# > route: /group/dataset# > attrs: {'c': 3, 'd': 4, 'e': array([[1., 1., 1.],#        ...,#        [1., 1., 1.]]), 'f': array([[1., 1., 1.],#        ...,#        [1., 1., 1.]])}# > shape: (20, 30, 10):cake:[EXTRA]Define required attributesDefine required attributes for all groups and all datasets:obj=BasicH5File('myfile.h5',group_attrs_required=('a','b'),dataset_attrs_required=('c','d'))As such, if a file does not comply, an assertion error is raised.
A file may have more attributes than those required, but no less.This feature is useful for ensuring that the HDF5 files to be read comply with the desired criteria.
One can even subclass theBasicH5Filefor easier use:classStrictH5File(BasicH5File):def__init__(self,path:str):super().__init__(path,group_attrs_required=('a','b'),dataset_attrs_required=('c','d'))obj=StrictH5File('myfile.h5')Additional notessimple_h5pyis not meant to be ah5pyreplacement but a useful sidekick.
Indeed, for "simple" use-cases, such as those shown here,simple_h5pyallows faster development byhidingmany of the implementation details.
As such,h5pyshould be employed for more advanced or custom needs.
Moroever, a file can be written withsimple_h5pyand then be read usingh5py, and viceversa.Equivalent snippet usingh5pyThe following code performs has the same effect as the sample snippet.Note the simpler syntax that allowssimple_h5py!importh5pyimportnumpyasnpgroup_attrs=dict(a=1,b=2)dataset=np.ones((5,4,3))dataset_attrs=dict(new=5,huge=np.ones((1000000,3)))# Use context manager to avoid open/closewithh5py.File('demo.h5','w')asobj:# Create groupobj.create_group(name='my_group')# Add attributes to group one at a timefork,vingroup_attrs.items():obj['my_group'].attrs[k]=v# Create datasetobj['my_group'].create_dataset('my_dataset',data=dataset)# Add attributes to dataset one at a timefork,vindataset_attrs.items():# Use try/except for capturing the "large attributes"try:obj['my_group']['my_dataset'].attrs[k]=vexceptBaseException:# Create an auxiliary dataset called in a helper group 'big_attrs'if'big_attrs'notinobj:obj.create_group(name='big_attrs')obj['big_attrs'].create_dataset('huge_attr',data=dataset_attrs['huge'])# Store the reference to the auxiliary datasetobj['my_group']['my_dataset'].attrs[k]=\obj['big_attrs']['huge_attr'].ref# Use context manager to avoid open/closewithh5py.File('demo.h5','r')asobj:# Read attributes and convert to dictionary for further useread_group_attrs=dict(obj['my_group'].attrs)read_dataset_attrs=dict(obj['my_group']['my_dataset'].attrs)# Read entire datasetread_dataset=obj['my_group']['my_dataset'][:]# Verify if any reference is present in the attributes (i.e. big attribute)fork,vinread_dataset_attrs.items():ifisinstance(v,h5py.Reference):read_dataset_attrs[k]=obj[v][:]# Display contentsprint(read_group_attrs)print(read_dataset_attrs)print(read_dataset[0])