Package: h5rdmtoolbox
HDF5 Research Data Management ToolboxNote, that the project is still under development!The "HDF5 Research Data Management Toolbox" (h5RDMtoolbox) is a python package supporting everybody who is working with
HDF5 to achieve a sustainable data lifecycle which follows
theFAIR (Findable, Accessible, Interoperable, Reusable)principles. It specifically supports the five main steps ofplanning,collecting,analyzing,sharingandreusingdata. Please visit thedocumentationfor detailed
information of try thequickstart using colab.HighlightsCombining HDF5 andxarrayto allow easy access to metadata and data during
analysis and processing (
seehere.Assigningmetadata with "globally unique and persistent identifiers"as required
byF1 of the FAIR principles. This "remove[s] ambiguity in the meaning of your published data...".Define standard attributes throughconventionsand enforce users to use
themUpload HDF5 files directly
torepositorieslikeZenodooruse them with noSQL databaseslikemongoDB.Who is the package for?For everybody, who is...... looking for a management approach for his or her data... community has not yet established a stable convention... working with small and big data, that fits into HDF5 files... looking for an easy way to work with HDF5, especially through Jupyter Notebooks... looking to integrate HDF5 with repositories and databases... looking for a way to do all the above whiles not needing to learn a new syntax... new to HDF5 and wants to learn about it, especially with respect to the FAIR principles and data managementWho is it not for?For everybody, who ...... is looking for a management approach which at the same time allows high-performance and/or parallel work with HDF5... has established conventions and managements approaches in his or her communityPackage Architecture/structureThe toolbox implements five modules, which are shown below. The numbers reference to their main usage in the stages in
the data lifecycle above. Except the wrapper module, which uses the convention module, all other modules are independent
of each other.Current implementation highlights in the modules:Thewrappermodule adds functionality on top of theh5pypackage. It allows to include so-called standard names,
which are defined in conventions. And it implements an interface with the packagexarray, which allows to carry
metadata from HDF5 to the user.For thedatabasemodule,hdfDBandmongoDBare implemented. ThehdfDBmodule allows to use HDF5 files as a
database. ThemongoDBmodule allows to use mongoDB as a database by mapping the metadata of HDF5 files to the
database.For therepositorymodule, a Zenodo interface is implemented. Zenodo is a repository, which allows to upload and
download data with a persistent identifier.For theconventionmodule,
thestandard attributesare implemented.Thelayoutmodule allows to define expectations on the internal layout (object names, location, attributes,
properties) of HDF5 files.QuickstartA quickstart notebook can be tested by clicking on the following badge:DocumentationPlease find a comprehensive documentation with many exampleshereor
by click on the image, which shows the research data lifecycle in the center and the respective toolbox features on the
outside:A paper is published in the journalinggrid.InstallationUse python 3.8 or higher (automatic testing is performed until 3.12). If you are a regular user, you can install the
package via pip:pip install h5RDMtoolboxInstall from source:Developers may clone the repository and install the package from source. Clone the repository first:git clone https://github.com/matthiasprobst/h5RDMtoolbox.git@mainThen, runpip install h5RDMtoolbox/Add--userif you do not have root access.For development installation runpip install -e h5RDMtoolbox/DependenciesThe core functionality depends on the following packages. Some of them are for general management others are very
specific to the features of the package:General dependencies are ...numpy>=1.20: Scientific computing, handling of arraysmatplotlib>=3.5.2: Plottingappdirs>=1.4.4: Managing user and application directoriespackaging: Version handlingIPython>=8.4.0: Pretty display of data in notebooksregex>=2020.7.9: Working with regular expressionsSpecific to the package are ...h5py=3.7.0: HDF5 file interfacexarray>=2022.3.0: Working with scientific arrays in combination with attributes. Allows carrying metadata from HDF5
to userpint>=0.19.2: Allows working with unitspint_xarray>=0.2.1: Working with units for usage with xarraypython-forge==18.6.0: Used to update function signatures when using
thestandard attributespyyaml>6.0.0: Reading and writing of yaml files, e.g. metadata definitions (conventions). Note, lower versions
collide with python 3.11requests: Used to download files from the internet or validate URLs, e.g. metadata definitions (conventions)Optional dependenciesTo run unit tests or to enable certain features, additional dependencies must be installed.Install optional dependencies by specifying them in square brackets after the package name, e.g.:pip install h5RDMtoolbox[mongodb][mongodb]pymongo>=4.2.0: Database solution for HDF5 files[csv]pandas>=1.4.3: Mainly used for reading csv and pretty printing[snt]xmltodict: Reading of xml filestabulate>=0.8.10: Pretty printing of tablespython-gitlab: Access to gitlab repositoriespypandoc>=2.3: Conversion of markdown files to htmlCiting the packageIf you intend to use the package in your work, you may cite the paper in the
journalinggridHere's the bibtext to it:@article{probst2023h5rdmtoolbox,
  title={h5RDMtoolbox-A Python Toolbox for FAIR Data Management around HDF5},
  author={Probst, Matthias and Pritz, Balazs},
  year={2023},
  publisher={ing. grid Preprint Repository}
}ContributionFeel free to contribute. Make sure to writedocstringsto your methods and classes and please write tests and use PEP
8 (https://peps.python.org/pep-0008/)Please write tests for your code and put them into thetest/folder. Visit theREADME filein the
test-folder for more information.Pleas also add a jupyter notebook in thedocs/folder in order to document your code. Please visit
theREADME filein the docs-folder for more information on how to compile the documentation.Please use thenumpy style for the docstrings:https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_numpy.html#example-numpy