Package: ch5mpy
Ch5mpyPronounced "champy".
This library provides a set of helper tools for easily reading or writing even complex objects to h5 files using the h5py library. It implements wrappers around h5py objects providing APIs identical to regular Python lists and dicts and to numpy ndarrays.See the complete documentation athttps://ch5mpy.readthedocs.io/en/latest/for more details.DescriptionCh5mpy provides a set of abstractions over h5py's (https://docs.h5py.org/en/stable/) objects for handling them as more commonly used objects :H5Dict: an object behaving as regular Python dictionaries, for exploring Files and Groups.H5List: an object behaving as regular Python lists for storing any set of objects.H5Array: an object behaving as Numpy ndarrays for dealing effortlessly with Datasets while keeping the memory usage low. This works by applying numpy functions to small chunks of the whole Dataset at a time.AttributeManager: a dict-like object for accessing an h5 object's metadata.read/write utily functions for effortlessly storing any object to an h5 file.Pickling has also been added to base h5 objects.PicklingCh5mpy provides Datasets, Groups and Files objects wrapping the h5py's equivalents to allow pickling. Those objects can be directly imported fromch5mpy:>>>fromch5mpyimportFile>>>fromch5mpyimportGroup>>>fromch5mpyimportDatasetTheH5Modeenum lists valid modes for opening an h5 file:>>>fromch5mpyimportH5ModeclassH5Mode(str,Enum):READ="r"# Readonly, file must existREAD_WRITE="r+"# Read/write, file must existWRITE_TRUNCATE="w"# Create file, truncate if existsWRITE="w-"# Create file, fail if existsREAD_WRITE_CREATE="a"# Read/write if exists, create otherwiseAttributesMetadata on Datasets, Groups and Files can be obtained and modified through the.attrsattribute, returning anAttributeManagerobject.AttributeManagersbehave like Python dictionaries for getting and setting any value.>>>fromch5mpyimportFile>>>f=File('some/file.h5')>>>f.attrsAttributeManager{value:1,creation:'02/08/2021',parent:None}>>>f.attrs['value']1AttributeManagerscorrectly handleNonevalues.H5DictAnH5Dictallows to explore the content of an H5 File or Group as if it was a regular Python dict. Any value can be set in anH5Dict. However, keys in anH5Dictare not loaded into memory until they are directly requested.Datasetsare wrapped and accessed asH5Arrays(see sectionH5Arrays).To create anH5Dict, aFileorGroupobject must be provided as argument:>>>fromch5mpyimportFile>>>fromch5mpyimportH5Dict>>>fromch5mpyimportH5Mode>>>>>>dct=H5Dict(File("dict.h5",H5Mode.READ_WRITE))>>>dctH5Dict{a:1,b:H5Array([1,2,3],shape=(3,),dtype=int64),c:{...}}Here,dctis anH5Dictwith 3 keysa, b and cwhere :amaps to the value1bmaps to a 1D Datasetcmaps to a sub H5Dict with keys and values not loaded yetAlternatively, anH5Dictcan be created directly from a path to an h5 file:>>>H5Dict.read("dict.h5")H5Dict{a:1,b:H5Array([1,2,3],shape=(3,),dtype=int64),c:{...}}H5ListAnH5Listbehave as regular Python lists, allowing to store and access any kind of object in an h5 file.H5Listsare usually created when regular lists are stored in an h5 file.As forH5Dicts,H5Listscan be created by providing aFileor by calling the.read()method:>>>fromch5mpyimportFile>>>fromch5mpyimportH5List>>>fromch5mpyimportH5Mode>>>>>>lst=H5List(File("backed_list.h5",H5Mode.READ_WRITE))>>>lstH5List[1.0,2,'4.']classO_:def__init__(self,v:float):self._v=vdef__repr__(self)->str:returnf"O({self._v})">>>lst.append(O(5.0))>>>lstH5List[1.0,2,'4.',O(5.0)]H5Listscan store regular integers, floats and strings, but can also store any object (such as theOobject at index 3 in this example).H5ArrayH5ArrayswrapDatasetsand implement numpy ndarrays' interface to behave as numpy ndarrays while controlling the amount of RAM used. The maximum amount of available RAM for performing operations can be set with the functionset_options(max_memory_usage=...), using suffixesB,K,MandGfor expressing amounts in bytes.H5Arrays can be created by passing aDatasetas argument.>>>fromch5mpyimportFile>>>fromch5mpyimportH5Mode>>>fromch5mpyimportH5Array>>>h5_array=H5Array(File("h5_arrays",H5Mode.READ_WRITE)["integers"])>>>h5_arrayH5Array([[0,1,2],[3,4,5],[6,7,8]],shape=(3,3),dtype=int64)>>>h5_array=H5Array(File("h5_arrays",H5Mode.READ_WRITE)["strings"])>>>h5_arrayH5Array(['blue','red','yellow'],shape=(3,),dtype='<U6')Then, all usual numpy indexing and methods can be used. To keep the memory footprint small, those methods will be applied repeatedly on small chunks of the underlying Dataset.To load an H5Array into memory as a numpy array, simply run :np.array(h5_array)Read/write utilitiesFunctionsTo store any array-like object (object which could be converted to a numpy ndarray), functionswrite_dataset()andwrite_datasets()respectively allow to store one or many such objects.To store any other object, call functionswrite_object()andwrite_objects().
To dertermine how the object will be stored in the h5 file, the following rules are applied:objects implementing theStoring APIwill be stored by calling the__h5_write__()functionobjects that can be converted to numpy arrays will be saved by callingwrite_dataset()numbers and strings will be stored directlyall other objects will be stored as binary data by first pickling themStoring APITo define by hand how an object is stored and read from an h5 file, you can implement the__h5_write__()and__h5_read__()methods:classYourObject:...def__h5_write__(self,values:ch5mpy.H5Dict[Any])->None:...@classmethoddef__h5_read__(cls,values:ch5mpy.H5Dict[Any])->YourObject:...Both__h5_write__()and__h5_read__()receive as input anH5Dictin which to store or retreive your object. Please note that__h5_read__()is a classmethod, called asYourObject.__h5_read__()and which is responsible for both reading data from theH5Dictand reconstructing an instance ofYourObject.RoadmapNumpy methods to implement forH5Arrays:Logic functionsnp.allnp.anynp.isfinitenp.isinfnp.isnannp.isnatnp.isneginfnp.isposinfnp.iscomplexnp.iscomplexobjnp.isfortrannp.isrealnp.isrealobjnp.isscalarnp.logical_andnp.logical_ornp.logical_notnp.logical_xornp.allclosenp.isclosenp.array_equalnp.array_equivnp.greaternp.greater_equalnp.lessnp.less_equalnp.equalnp.not_equalBinary operationsnp.bitwize_andnp.bitwize_ornp.bitwize_xornp.invertnp.left_shiftnp.right_shiftnp.packbitsnp.unpackbitsnp.binary_reprString operationsnp.char.addnp.char.multiplynp.char.modnp.char.capitalizenp.char.centernp.char.decodenp.char.encodenp.char.expandtabsnp.char.joinnp.char.ljustnp.char.lowernp.char.lstripnp.char.partitionnp.char.replacenp.char.rjustnp.char.rpartitionnp.char.rsplitnp.char.rstripnp.char.splitnp.char.splitlinesnp.char.stripnp.char.swapcasenp.char.titlenp.char.translatenp.char.uppernp.char.zfillnp.char.equalnp.char.not_equalnp.char.greater_equalnp.char.less_equalnp.char.greaternp.char.lessnp.char.compare_chararraysnp.char.countnp.char.endswithnp.char.findnp.char.indexnp.char.isalphanp.char.isalnumnp.char.isdecimalnp.char.isdigitnp.char.islowernp.char.isnumericnp.char.isspacenp.char.istitlenp.char.isuppernp.char.rfindnp.char.rindexnp.char.startswithnp.char.str_lennp.char.arraynp.char.asarraynp.char.chararrayMathematical functionsnp.sinnp.cosnp.tannp.arcsinnp.arccosnp.arctannp.hypotnp.arctan2np.degreesnp.radiansnp.unwrapnp.deg2radnp.rad2degnp.sinhnp.coshnp.tanhnp.arcsinhnp.arccoshnp.arctanhnp.aroundnp.rintnp.fixnp.floornp.ceilnp.truncnp.prodnp.sumnp.nanprodnp.nansumnp.cumprodnp.cumsumnp.nancumprodnp.nancumsumnp.diffnp.ediff1dnp.gradientnp.crossnp.trapznp.expnp.expm1np.exp2np.lognp.log10np.log2np.log1pnp.logaddexpnp.logaddexp2np.i0np.sincnp.signbitnp.copysignnp.frexpnp.ldexpnp.nextafternp.spacingnp.lcmnp.gcdnp.addnp.reciprocalnp.positivenp.negativenp.multiplynp.dividenp.powernp.subtractnp.true_dividenp.floor_dividenp.float_powernp.fmodnp.modnp.modfnp.remaindernp.divmodnp.anglenp.realnp.imagnp.conjnp.conjugatenp.maximumnp.fmaxnp.amaxnp.nanmaxnp.minimumnp.fminnp.aminnp.nanminnp.convolvenp.clipnp.sqrtnp.cbrtnp.squarenp.absolutenp.fabsnp.signnp.heavisidenp.nan_to_numnp.real_if_closenp.interpSet routinesnp.uniquenp.in1dnp.intersect1dnp.isinnp.setdiff1dnp.setxor1dnp.union1dArray creation routinesnp.emptych5mpy.emptynp.empty_likenp.eyenp.identitynp.onesch5mpy.onesnp.ones_likenp.zerosch5mpy.zerosnp.zeros_likenp.fullch5mpy.fullnp.full_likenp.arraynp.asarraynp.asanyarraynp.ascontiguousarraynp.asmatrixnp.copynp.frombuffernp.from_dlpacknp.fromfilenp.fromfunctionnp.fromiternp.fromstringnp.loadtxtnp.core.records.arraynp.core.records.fromarraysnp.core.records.fromrecordsnp.core.records.fromstringnp.core.records.fromfilenp.core.defchararray.arraynp.core.defchararray.asarraynp.arangenp.linspacenp.logspacenp.geomspacenp.meshgridnp.mgridnp.ogridnp.diagnp.diagflatnp.trinp.trilnp.triunp.vandernp.matnp.bmatArray manipulation routinesnp.copytonp.shapenp.reshapenp.ravelnp.ndarray.flatnp.ndarray.flattennp.moveaxisnp.rollaxisnp.swapaxesnp.ndarray.Tnp.transposenp.atleast_1dnp.atleast_2dnp.atleast_3dnp.broadcastnp.broadcast_tonp.broadcast_arraysnp.expand_dimsnp.squeezenp.asarraynp.asanyarraynp.asmatrixnp.asfarraynp.asfortranarraynp.ascontiguousarraynp.asarray_chkfinitenp.requirenp.concatenatenp.stacknp.blocknp.vstacknp.hstacknp.dstacknp.column_stacknp.row_stacknp.splitnp.array_splitnp.dsplitnp.hsplitnp.vsplitnp.tilenp.repeatnp.deletenp.insertnp.appendnp.resizenp.trim_zerosnp.uniquenp.flipnp.fliplrnp.flipudnp.reshapenp.rollnp.rot90Sorting, searching, and countingnp.sortnp.lexsortnp.argsortnp.ndarray.sortnp.sort_complexnp.partitionnp.argpartitionnp.argmaxnp.nanargmaxnp.argminnp.nanargminnp.argwherenp.nonzeronp.flatnonzeronp.wherenp.searchsortednp.extractnp.count_nonzeroRandombetabinomialbyteschisquarechoicedirichletexponentialfgammaget_stategeometricgumbelhypergeometriclaplacelogisticlognormallogseriesmultinomialmultivariate_normalnegative_binomialnoncentral_chisquarenoncentral_fnormalparetopermutationpoissonpowerrandrandintrandnrandomrandom_integersrandom_samplerayleighseedset_stateshufflestandard_cauchystandard_exponentialstandard_gammastandard_normalstandard_ttriangularuniformvonmiseswaldweibullzipf