Package: h5torch
h5torchHDF5 data utilities for PyTorch.h5torchconsists of two main parts: (1)h5torch.File: a wrapper aroundh5py.Fileas an interface to create HDF5 files compatible with (2)h5torch.Dataset, a wrapper aroundtorch.utils.data.Dataset. As a library,h5torchestablishes a "code" for linkingh5pyandtorch. To do this, this package has to formulate a vocabulary for how datasets generally look, unifying as many ML settings to the best of its abilities. In turn, this vocabulary allows dataloading of various machine learning data settings from a single dataset class definition, reducing boilerplate in your projects.Who is this package for?Loading data from HDF5 files allows for efficient data-loading from anon-diskformat, drastically reducing memory overhead. Additionally, you will find your datasets to be moreorganizedusing the HDF5 format, as everything is neatly arrayed in a single file.If you want to use this package but are not sure your use-case is covered by the current formulation of the package, feel free to open an issue.InstallSince PyTorch is a dependency ofh5torch, we recommendinstalling PyTorchindependently first, as your system may require a specific version (e.g. CUDA drivers).After PyTorch installation,h5torchcan be installed usingpippipinstallh5torchPackage conceptsStoringThe main idea behindh5torchis that datasets can usually be formulated as being aligned to a central object. E.g. in a classical supervised learning setup, features/inputs are aligned to a label vector/matrix. In recommender systems, a score matrix is the central object, with features aligned to rows and columns.h5torchallows creating and reading HDF5 datasets for use in PyTorch using this dogma. When creating a new dataset, the first data object that should be registered is thecentralobject. The type ofcentralobject is flexible:N-D: for regular dense data. The number of dimensions in this object will dictate how many possible aligned axes can exist.coo: The sparse variant asN-D. The number of dimensions here can be arbitrary high.csr: For sparse 2D arrays, this central data type can only have 2 aligned axes and can only be sampled along the first dimensionvlen: For variable length 1D arrays. This central data type can only have one aligned axis (0).separate: For objects that are better stored in separate groups instead of as one dataset. An example is variable shape N-D objects such as variably-sized images. This central data type can only have one aligned axis (0).Along this central object, axis objects can be aligned. The first dimension length of any axis object must correspond to the length of the central data object to that dimension. For example, a central data object of shape (50, 40) can only have 50-length and 40-length objects aligned to its first and second axis, respectively. For axis objects, these possibilities are available:N-D: Can have arbitrary number of dimensions. E.g. equally-sized images:(N, 3, H, W).csr: Max 2 dimensions, rows will be sampled. E.g. A sparse scRNA-seq count matrixvlen: Variable length 1D arrays. E.g. Tokenized text as variable length arrays of integers.separate: For objects that are better stored in separate groups instead of as one dataset. An example is variable shape N-D objects such as variably-sized images.Note there is no support forcoodata type for aligned objects, that is because aligned axis objects require efficient indexing along their first dimension.Also note that there is no limit on the number of data objects aligned to an axis. For example, in the case of images aligned to a central label vector, extra information of every image can be added such as the URL, the date said image was taken, the geolocation of that image, ...Besides the central and axis objects, you can also storeunstructureddata which can be any length or dimension and follow any of the above-mentioned data types (includingcoo). This could for example be a vocabularium vector or the names of classes...SamplingOnce a dataset is created usingh5torch.File, it can be used as a PyTorch Dataset usingh5torch.Dataset. Sampling can occur along any of the axes in the central object, upon which the corresponding indices in the objects aligned to that axis are also sampled. Alternatively,coosampling (available forN-Dandcoo-type central objects) samples one specific element of the central dataset, along with the corresponding indices of all axis-aligned objects.UsageRefer to thetutorial on the documentation page.Package roadmapImplement typingProvide data type conversion capabilities for registering datasetsAdd support for custom samplersAdd support for making data splitsAdd a slice samplerImplement a way to pre-specify dataset size and append to itAdd better docsAdd testsImplement a collater for variable length objectsBenchmarks