Package: lyncs-io
I/O functions for Python and LQCD file formatsLyncs IO offers two high-level functionsloadandsave(ordumpas alias ofsave).The main features of this module areSeamlessly IO, reading and writing made simple.
In most of the cases, after savingsave(obj, filename),
loadingobj=load(filename)returns the original Python object.
This feature is already ensured by formats likepickle, but we
try to ensure it as much as possible also for other formats.Many formats supported. The file format can be specified either
via the filename's extension or with the optionformatpassed toload/save. The structure of the package is flexible enough to
easily accomodate new/customized file formats as these arise.
See [Adding a file format] for guidelines.Support for archives. In case of archives, e.g. HDF5, zip etc.,
the content can be accessed directly by specifying it in the path.
For instance withdirectory/file.h5/content,directory/file.h5is the file path, and the remaining is content to be accessed that
will be searched inside the file (this is inspired byh5py).Support for Parallel IO. Where possible and implemented,
parallel IO is supported. This is enabled either via MPI providing
a valid communicator with the optioncomm, or viaDaskproviding the optionchunks(seeDask's Array).Omission of extension. When saving, if the extension is omitted,
the optimal file format is deduced from the data type and the extension
is added to the filename. When loading, any extension is considered,
i.e.filename.*, and if only one match is available, the file is loaded.InstallationThe package can be installed viapip:pipinstall[--user]lyncs_ioNOTE: for enabling parallel IO, lyncs_io requires a working MPI installation.
This can be installed viaapt-get:sudoapt-getinstalllibopenmpi-devopenmpi-binOR usingconda:condainstall-canacondampi4pyParallel IO can then be enabled viapipinstall[--user]lyncs_io[mpi]DocumentationThe package provides three high-level functions:head: loads the metadata of a file (e.g.shape,dtype, etc)load: loads the content of a filesaveordump: saves data to fileimportnumpyasnpimportlyncs_ioasioarr1=np.random.rand(10,10,10)io.save(arr1,"data.npy")arr2=io.load("data.npy")assert(arr1==arr2).all()NOTE: forsavewe use the orderdata, filename. This is the opposite
of what done innumpybut consistent withpickle'sdump. This order
is preferred because the function can be used directly as a method
for a class sinceself, i.e. thedata, would be passed as the first
argument ofsave.Supported file formatsFormatExtensionsBinaryArchiveParallel MPIParallel DaskpicklepklyesnononodilldllyesnononoJSONjsonnonononoASCIItxtnonononoNumpynpyyesnoyesyesNumpyznpzyesyesTODOTODOHDF5hdf5,h5yesyesyesTODOlimelimeyesTODOyesyesTartar, tar.*-yesyesnoopenqcdoqcdyesnoTODOTODOIO with HDF5importnumpyasnpimportlyncs_ioasioarr1=np.random.rand(10,10,10)io.save(arr1,"data.h5/random")arr2=np.zeros_like(arr1)io.save(arr2,"data.h5/zeros")arrs=io.load("data.h5")assert(arr1==arrs["random"]).all()assert(arr2==arrs["zeros"]).all()Also the content of nested dictionary can be stored with HDF5:importnumpyasnpimportlyncs_ioasiomydict={"random":{"arr0":np.random.rand(10,10,10),"arr1":np.random.rand(5,5),},"zeros":np.zeros((4,4,4,4)),}# once a dictionary (or mapping) is passed it is written as a groupio.save(mydict,"data.h5")# all the datasets in the .h5 file are loaded here using all_data argumentloaded_dict=io.load("data.h5",all_data=True)assert(mydict["random"]["arr0"]==loaded_dict["random"]["arr0"]).all()assert(mydict["random"]["arr1"]==loaded_dict["random"]["arr1"]).all()assert(mydict["zeros"]==loaded_dict["zeros"]).all()Parallel IO via MPI can be enabled with a parallel installation of HDF5.
For doing so, please refer tohttps://docs.h5py.org/en/stable/mpi.html.IO with MPIimportnumpyasnpimportlyncs_ioasiofrommpi4pyimportMPI# Assume 2D cartesian topologycomm=MPI.COMM_WORLDdims=(2,2)# e.g. 4 procscartesian2d=comm.Create_cart(dims=dims)oarr=np.random.rand(6,4,2,2)io.save(oarr,"pario.npy",comm=cartesian2d)iarr=io.load("pario.npy",comm=cartesian2d)assert(iarr==oarr).all()NOTE: Parallel IO is enabled once a valid cartesian communicator is passed toloadorsaveroutines, otherwise Serial IO is performed. Currently onlynumpyformat supports this functionality.IO with Daskimportlyncs_ioasioimportdaskfromdistributedimportClient,progressclient=Client(n_workers=2,threads_per_worker=1)x=da.arange(0,128).reshape((16,8)).rechunk(chunks=(8,4))xout_lazy=io.save(x,"pario.npy")xin_lazy=io.load("pario.npy",chunks=(8,4))assert(x.compute()==xin_lazy.compute()).all()client.shutdown()NOTE: Parallel IO with Dask is enabled once a valid chunk size is passed toloadroutine usingchunksparameter. Forsaveroutine, the DaskIO is enabled only if the array passed is a Dask Array. Currently onlynumpyformat supports this functionality.IO with Tarimportnumpyasnpimportlyncs_ioasioarr1=np.random.rand(10,10,10)io.save(arr1,"data.tar/random.npy")arr2=np.zeros_like(arr1)io.save(arr2,"data.tar/zeros.npy")arrs=io.load("data.tar")assert(arr1==arrs["random.npy"]).all()assert(arr2==arrs["zeros.npy"]).all()Also the content of nested dictionary can be stored with Tar:mydict={"random":{"arr0.npy":np.random.rand(10,10,10),"arr1.npy":np.random.rand(5,5),},"zeros.npy":np.zeros((4,4,4,4)),}io.save(mydict,'data.npy')loaded_dict=io.load('data.npy',all_data=True)assert(mydict["random"]["arr0.npy"]==loaded_dict["random"]["arr0.npy"]).all()assert(mydict["random"]["arr1.npy"]==loaded_dict["random"]["arr1.npy"]).all()assert(mydict["zeros.npy"]==loaded_dict["zeros.npy"]).all()Note:Some formats inside a Tar are not currently supported. (SeeIssues)When loading/saving a file in series, it's done directly on the memory. When in parallel, files are first written on the secondary storage before being saved/loaded.Adding a file formatNew file formats can be registered providing providing where possible the respectivehead,loadandsavefunctions.
For example thepicklefile format can be registered as follow:importpicklefromlyncs_io.formatsimportregisterregister("pickle",# Name of the formatextensions=["pkl"],# List of extensionshead=None,# Function called by head (omitted)load=pickle.load,# Function called by loadsave=pickle.dump,# Function called by savedescription="Pickle file format",# Short description)AcknowledgmentsAuthorsSimone Bacchio (sbacchio)Christodoulos Stylianou (cstyl)Alexandros Angeli (alexandrosangeli)FundingsPRACE-6IP, Grant agreement ID: 823767, Project name: LyNcs.