Package: dareblopy
Framework agnostic, faster data reading for DeepLearning.A native extension for Python built with C++ andpybind11.Installation•Why?•What is the performance gain?•Tutorial•LicenseDataReadingBlocks forPython is a python module that provides collection of C++ backed data reading primitives.
It targets deep-learning needs, but it is framework agnostic.InstallationAvailable as pypi package:$ pip install dareblopyTo build from sources refer towiki page.Why?Development initially started to speedup reading fromZIParchives, reduce copying data, increase time ofGILbeing released to improve concurrency.But why reading fromZIParchive?Reading a ton of small files (which is often the case) can be slow, specially if the drive is network attached, e.g. withNFS. However, the bottle neck here is hardly the disk speed, but the overhead of filesystem, name-lookup, creating file descriptors, and additional network usage if NFS is used.If, all the small files are agglomerated into larger file (or several large files), that improves performance substantially. This is exactly the reason behindTFRecordsinTensorFlow:To read data efficiently it can be helpful to serialize your data and store it in a set of files (100-200MB each) that can each be read linearly. This is especially true if the data is being streamed over a network. This can also be useful for caching any data-preprocessing.The downside ofTFRecordsis that it'sTensorFlowonly.A much simpler, yet still effective solution is to store data inZIParchive with zero compression. However, usingzipfilepackage from standard library can be slow, since it is implemented purely inPythonand in certain cases can cause unnecessary data copying.That's precisely the reason behind development ofDareBlopy. In addition to that it also has such features as:Readying JPEG images directly to numpy arrays (fromZIPand from filesystem), to reduce memory usage and unnecessary data copies.Two JPEG backends selectable at run-time:libjpegandlibjpeg-turbo. Both backends are embedded intoDareBlopyand do not depend on any system package.Reading ofTFRecords(not all features are support though) without dependency onTensorFlowthat enables usage of datasets stored asTFRecordswith ML frameworks other thanTensorFlow, e.g.Pytorch.Random yielders,  iterators and, dataloaders to simplify doing DataLearning withTFRecordswith other ML frameworks.No dependency on system packages. You install it from pip - it works.Support for compressedZIParchives, includingLZ4compression.Virtual filesystem. Allowsmountingof zip archives.What is the performance gain?Well, it depends a lot on a particular use-case. Let's consider several. All details of the benchmarks you can find inrun_benchmark.py. You can also run it on your machine and compare results to the ones reported here.Reading files tobytesPython'sbytesobject can be a bit nasty. Generally speaking, you can not return from C/C++ land data as abytesobject without making a data copy. That's because memory forbytesobject must be allocated as one chunk for both, the header and data itself. InDareBlopythis extra copy is eliminated, you can find detailshere.In this test scenario, we read 200 files, each of which ~30kb. Reading is done from local filesystem and from aZIParchive.Reading files usingDareBlopyis faster even when read from filesystem, but when read fromZIPit provides substantial improvement.Reading JPEGs tonumpy'sndarrayThis is whereDareBlopy's feature of direct readying tonumpyarray is demonstrated. When the file is read, it is decompressed directly to a preallocated numpy array, and all of that happens on C++ land whileGILis released.Note: here PIL v.7.0.0 is used, on Ubuntu 18. In my installation, it does not uselibjpeg-turbo.It this case, difference betweenZIP/filesystem is quite insignificant, but things change dramatically if filesystem is streamed over a network:Reading TFRecordsDareBlopycan readTensorFlowrecords. This functionality was developed in the first place for reading FFHQ dataset from TFRecords.It introduces alias tostringtype:uint8, which allows to return directlynumpyarray if the shape is known beforehand.For example, code like:features={'data':db.FixedLenFeature([],db.string)}Can be replaced with:features={'data':db.FixedLenFeature([3,32,32],db.uint8)}This decoding to numpy array comes at zero cost, which is demonstrated below:TutorialImport DareBlopyimportdareblopyasdbfromIPython.displayimportImage,displayimportPIL.ImageOpen zip archive:archive=db.open_zip_archive("test_utils/test_image_archive.zip")Read image to bytes and display:b=archive.open_as_bytes('0.jpg')Image(b)Alternatively, read image to numpy:img=archive.read_jpg_as_numpy('0.jpg')img.shape(256, 256, 3)display(PIL.Image.fromarray(img))For more advanced usage please refer to:Tests:https://github.com/podgorskiy/DareBlopy/blob/master/test_dareblopy.pyBenchmark:https://github.com/podgorskiy/DareBlopy/blob/master/run_benchmark.pyAPI reference page:https://podgorskiy.github.io/DareBlopy/dareblopy.htmlLicenseApache License 2.0