Package: h5io-browser
Easy navigation and data storage for HDF5Thehierarchical data format (HDF)is aimed to ensure efficient and equitable access to
science and engineering data across platforms and environments. Theh5pypackage provides a
pythonic interface to the HDF5 binary data format and theh5iopackage simplifies this
interface by introducing theread_hdf5()andwrite_hdf5()functions for loading and storing python objects in HDF5.
Theh5iopackage also provides alist_file_contents()function to print the internal,
structure of an HDF5 file.Theh5io_browserpackage extends this interface by providing a pointerh5io_browser.Pointerto a specific path
inside the hierarchical structure of the HDF5 file. With this pointer, data can be read, stored, copied and deleted from
the HDF5 file, while at the same time simplifying the navigation inside the hierarchy of the file. Theh5io_browserpackage is developed with three constraints and goals:Simplify navigating HDF5 files created by theh5iopackage. This includes interactive
navigation inside an interactive Python shell or a Jupyter Notebook environment.Integrate standard functionality to interact with the data stored in the HDF5 file like read, write, copy and delete
using the interface defined by theh5pypackage and theh5iopackage.Finally, balance flexibility and performance. Just like theh5iopackage, theh5io_browseronly opens the HDF5 file when accessing the data and does not maintain an open file handle while
waiting for user input. At the same time the interface defined by theh5iopackage
is extended to store multiple python objects at the same time for improved performance.InstallationTheh5io_browserpackage can be installed either via thePython Package Index:pip install h5io_browserOr alternatively, via the community channel on theconda package managermaintained by theconda-forge community:conda install -c conda-forge h5io_browserExampleDemonstration of the basic functionality of theh5io_browsermodule.Import ModuleStart by importing theh5io_browsermodule:importh5io_browserashbFrom theh5io_browsermodule thePointer()object is created to access a new HDF5 file namednew.h5:hp=hb.Pointer(file_name="new.h5")Write DataFor demonstration three different objects are written to the HDF5 file:a list with the numbers one and two is stored in the HDF5 pathdata/a_listan integer number is stored in the HDF5 pathdata/an_integer_numbera dictionary is stored in the HDF5 pathdata/sub_path/a_dictionaryThis can either be done using the edge notation, known from accessing python dictionaries, or alternatively using thewrite_dict()function which can store multiple objects in the HDF5 file, while opening it only once.hp["data/a_list"]=[1,2]hp.write_dict(data_dict={"data/an_integer_number":3,"data/sub_path/a_dictionary":{"d":4,"e":5},})Read DataOne strength of theh5io_browserpackage is the support for interactive python environments like, Jupyter notebooks.
To browse the HDF5 file by executing thePointer()object:hpIn comparison the string representation lists thefile_name,h5_pathas well as thenodesandgroupsat thish5_path:str(hp)>>>'Pointer(file_name="/Users/jan/test/new.h5", h5_path="/") {"groups": ["data"], "nodes": []}'List content of the HDF5 file at the currenth5_pathusing thelist_all()function:hp.list_all()>>>['data']In analogy thegroupsandnodesof anyh5_patheither relative to the currenth5_pathor as absoluteh5_pathcan be analysed using thelist_h5_path():hp.list_h5_path(h5_path="data")>>>{'groups':['sub_path'],'nodes':['a_list','an_integer_number']}To continue browsing the HDF5 file the edge bracket notation can be used, just like it s commonly used for python
dictionaries to browse the HDF5 file:hp["data"].list_all()>>>['a_list','an_integer_number','sub_path']The object which is returned is again a Pointer with the updatedh5_path, which changed from/to/data:hp.h5_path,hp["data"].h5_path>>>('/','/data')Finally, individual nodes of the HDF5 file can be loaded with the same syntax using the/notation known from the
file system, or by combining multiple edge brackets:hp["data/a_list"],hp["data"]["a_list"]>>>([1,2],[1,2])Convert to DictionaryTo computationally browse through the contents of an HDF5 file, theto_dict()method extends the interactive browsing
capabilities. By default it returns a flat dictionary with the keys representing theh5_pathof the individual nodes
and the values being the data stored in these nodes. Internally, this loads the whole tree structure, starting from the
currenth5_path, so depending on the size of the HDF5 file this can take quite some time:hp.to_dict()>>>{'data/a_list':[1,2],>>>'data/an_integer_number':3,>>>'data/sub_path/a_dictionary':{'d':4,'e':5}}An alternative representation, is the hierarchical representation which can be enabled by thehierarchicalbeing set
toTrue. Then the data is represented as a nested dictionary:hp.to_dict(hierarchical=True)>>>{'data':{'a_list':[1,2],>>>'an_integer_number':3,>>>'sub_path':{'a_dictionary':{'d':4,'e':5}}}}With StatementFor compatibility with other file access methods, theh5io_browserpackage also supports the with statement notation.
Still technically this does not change the behavior, even when opened with a with statement the HDF5 file is closed
between individual function calls.withhb.Pointer(file_name="new.h5")ashp:print(hp["data/a_list"])>>>[1,2]Delete DataTo delete data from an HDF5 file using theh5io_browserthe standard pythondelfunction can be used in analogy to
deleting items from a python dictionary. To demonstrate the deletion a new node is added nameddata/new/entry/test:hp["data/new/entry/test"]=4To list the node, theto_dict()function is used with thehierarchicalparameter to highlight the nested structure:hp["data/new"].to_dict(hierarchical=True)>>>{'entry':{'test':4}}The node is then deleted using thedelfunction. While this removes the node from the index the file size remains the
same, which is one of the limitations of the HDF5 format. Consequently, it is not recommended to create and remove nodes
in the HDF5 files frequently:print(hp.file_size())delhp["data/new/entry/test"]print(hp.file_size())>>>(18484,18484)Even after the deletion of the last node the groups are still included in the HDF5 file. They are not listed by theto_dict()function, as it recursively iterates over all nodes below the currenth5_path:hp["data/new"].to_dict(hierarchical=True)>>>{}Still with thelist_all()function lists all nodes and groups at a currenth5_pathincluding empty groups, like theentrygroup in this case:hp["data/new"].list_all()>>>['entry']To remove the group from the HDF5 file the samedelcommand is used:delhp["data/new"]After deleting both the newly created groups and their nodes the original hierarchy of the HDF5 file is restored:hp.to_dict(hierarchical=True)>>>{'data':{'a_list':[1,2],>>>'an_integer_number':3,>>>'sub_path':{'a_dictionary':{'d':4,'e':5}}}}Still even after deleting the nodes from the HDF5 file, the file size remains the same:hp.file_size()>>>18484Loop over NodesTo simplify iterating recursively over all nodes contained in the selectedh5_paththePointer()object can be used
as iterator:hp_data=hp["data"]{h5_path:hp_data[h5_path]forh5_pathinhp_data}>>>{'a_list':[1,2],>>>'an_integer_number':3,>>>'sub_path/a_dictionary':{'d':4,'e':5}}Copy DataIn addition to adding, browsing and removing data from an existing HDF5 file, thePointer()object can also be used to
copy data inside a given HDF5 file or copy data from one HDF5 file to another. A new HDF5 file is created, namedcopy.h5:hp_copy=hb.Pointer(file_name="copy.h5")The data is transferred from the existingPointer()object to the new HDF5 file using thecopy_to()functions:hp["data"].copy_to(hp_copy)hp_copyDisclaimerWhile we try to develop a stable and reliable software library, the development remains a opensource project under the
BSD 3-Clause License without any warranties:BSD 3-Clause License

Copyright (c) 2023, Jan Janssen
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

* Redistributions of source code must retain the above copyright notice, this
  list of conditions and the following disclaimer.

* Redistributions in binary form must reproduce the above copyright notice,
  this list of conditions and the following disclaimer in the documentation
  and/or other materials provided with the distribution.

* Neither the name of the copyright holder nor the names of its
  contributors may be used to endorse or promote products derived from
  this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.