Package: h5mapper
h5mapperh5mapperis a pythonic ORM-like tool for reading and writing HDF5 data.It is built on top ofh5pyand lets you define types of.h5 files as python classeswhich you can then easilycreate from raw sources(e.g. files, urls...),serve(use asDatasetfor aDataloader),
or dynamically populate (logs, checkpoints of an experiment).ContentInstallationQuickstartTypedFileFeatureExamplesDevelopmentLicenseInstallationpiph5mapperis on pypi, to install it, one only needs topipinstallh5mapperdeveloper installfor playing around with the internals of the package, a good solution is to firstgitclonehttps://github.com/ktonal/h5mapper.gitand thenpipinstall-eh5mapper/which installs the repo in editable mode.QuickstartTypedFileh5massumes that you want to store collections of contiguous arrays in single datasets and that you want several such concatenated datasets in a file.Thus,TypedFileallows you to create and read files that maintain a 2-d reference system, where contiguous arrays are stored within features and indexed by their source's id.Such a file might then look like<Experiment"experiment.h5">
---------------------------------------------------->sources'idsaxis|"planes/01.jpeg"|"train"|||data/||images/(32,32)|None|labels/(1,)|None|logs/||loss/None|(10000,)|...
V
featuresaxiswhere the entries correspond to the shapes of arrays or their absence (None).Note that this is a different approach than storing each file or image in a separate dataset.
In this case, there would be anh5py.Datasetlocated atdata/images/planes/01.jpegalthough in our
example, the only dataset is atdata/images/and one of its regions is indexed by the id"planes/01.jpeg"For interacting with files that follow this particular structure, simply define a classimporth5mapperash5mclassExperiment(h5m.TypedFile):data=h5m.Group(# your custom h5m.Feature classes:images=Image(),labels=DirLabels())logs=h5m.Group(loss=h5m.Array())create,addnow, create an instance, load data from files through parallel jobs and add data on the fly :# create instance from raw sourcesexp=Experiment.create("experiment.h5",# those are then used as ids :sources=["planes/01.jpeg","planes/02.jpeg"],n_workers=8)...# add id <-> data on the fly :exp.logs.add("train",dict(loss=losses_array))get,refsand__getitem__There are 3 main options to read data from aTypedFileor one of itsProxy1/ By their id>>exp.logs.get("train")Out:{"loss":np.array([...])}# which, in this case, is equivalent to>>exp.logs["train"]Out:{"loss":np.array([...])}# because `exp.logs` is a Group and Groups only support id-based indexing2/ By the index of their ids through theirrefsattribute :>>exp.data.images[exp.data.images.refs[0]].shapeOut:(32,32)Which works becauseexp.data.imagesis aDatasetand onlyDatasetshaverefs3/ with anyitemsupported by theh5py.Dataset>>exp.data.labels[:32]Out:np.array([0,0,....])Which only works forDatasets - not forGroups.Note that, in this last case, you are indexing into theconcatenation of all sub-arrays along their first axis.The same interface is also implemented forset(source, data)and__setitem__Featureh5mexposes a class that helps you configure the behaviour of yourTypedFileclasses and the properties of the .h5 they create.theFeatureclass helps you define :how sources' ids are loaded into arrays (feature.load(source))which types of files are supportedhow the data is stored byh5py(compression, chunks)which extraction parameters need to be stored with the data (e.g. sample rate of audio files)custom-methods relevant to this kind of dataOnce you defined aFeatureclass, attach it to the class dict of aTypedFile, that's it!For example :importh5mapperash5mclassMyFeature(h5m.Feature):# only sources matching this pattern will be passed to load(...)__re__=r".special$"# args for the h5py.Dataset__ds_kwargs__=dict(compression='lzf',chunks=(1,350))def__init__(self,my_extraction_param=0):self.my_extraction_param=my_extraction_param@propertydefattrs(self):# those are then written in the h5py.Group.attrsreturn{"p":self.my_extraction_param}defload(self,source):"""your method to get an np.ndarray or a dict thereoffrom a path, an url, whatever sources you have..."""returndatadefplot(self,data):"""custom plotting method for this kind of data"""# ...# attach itclassData(h5m.TypedFile):feat=MyFeature(47)# load sources...f=Data.create(....)# read your data through __getitem__batch=f.feat[4:8]# access your methodf.feat.plot(batch)# modify the file through __setitem__f.feat[4:8]=batch**2for more examples, checkouth5mapper/h5mapper/features.py.servePrimarly designed withpytorchusers in mind,h5mplays very nicely with theDatasetclass :classMyDS(h5m.TypedFile,torch.utils.data.Dataset):x=MyInputFeature(42)def__getitem__(self,item):returnself.x[item],self.labels[item]def__len__(self):returnlen(self.x)ds=MyDS.create("train.h5",sources,keep_open=True)dl=torch.utils.data.DataLoader(ds,batch_size=16,num_workers=8,pin_memory=True)TypedFileeven have a method that takes the Dataloader args and a batch object filled withBatchItemsand returns
a Dataloader that will yield such batch objects.Example :f=TypedFile("train.h5",keep_open=True)loader=f.serve(# batch object :dict(x=h5m.Input(key='data/image',getter=h5m.GetId()),labels=h5m.Target(key='data/labels',getter=h5m.GetId())),# Dataloader kwargs :num_workers=8,pin_memory=True,batch_size=32,shuffle=True)Examplesinh5mapper/examplesyou'll find for nowa train script with data, checkpoints and logs indataset_and_logs.pya script for benchmarking batch-loading times of different optionsDevelopmenth5mapperis just getting started and you're welcome to contribute!You'll find some tests you can run from the root of the repo with a simplepytestIf you'd like to get involved, just drop us an email :ktonalberlin@gmail.comLicenseh5mapperis distributed under the terms of the MIT License.