Package: thunderpack
⚡ ThunderPackBlazingly fast multi-modal data format for training deep neural networks❓ TL;DRMost deep learning comprise of media (image, video, sound) are distributed as plain individual files, but this can incur in a lot of inefficiencies due to the filesystem behaving poorly when many files are involved. ThunderPack solves the issue by usingLMDBan lightweight database, that supports blazingly fast reads.Benchmark of random read access on various data tensor storage solutions (lower read time is better). ThunderPack scales very well to large amount of datapoints, whereas other solutions present significant slowdowns even when using local SSD hardware.🌟 FeaturesOptimized read support- ThunderPack offers specialized read-only APIs that are orders of magnitude faster than common generic interfaces that also support writing.Concurrency Support- ThunderPack is thread safe and can be used withtorch.data.util.DataLoaderwithout issue.Memory mapping- Thanks to the use of LMDB, data is memory-mapped by default, drastically reducing the read times for entries already present in the filesystem cache.Data Locality- Data and labels are stored together, reducing read times.Immense Flexibility- Unlike other dataloader formats that trade off speed with usability, ThunderPack keeps a human-friendly dictionary-like interface that supports arbitrary sampling and that can be edited after creation.Improved Tensor I/O- Faster primitives for (de)serializing Numpyndarrays, TorchTensors andJaxndarrays.Extensive extension support- ThunderPack supports a wide variety of data formats out of the box.Customizability- ThunderPack can easily be extended to support other  (also feel free to submit a PR with your extension of choice).💾 InstallationOr for the latest version:pipinstallgit+https://github.com/JJGO/thunderpack.gitYou can alsomanuallyinstall it by cloning it, installing dependencies, and adding it to yourPYTHONPATHgitclonehttps://github.com/JJGO/thunderpack
python-mpipinstall-r./thunderpack/requirements.txtexportPYTHONPATH="$PYTHONPATH:$(realpath./thunderpack)"QuickstartThunderpack has asymetric APIs for writing and reading data to maximize read throughput and speed.First, we create a dataset using theThunderDBobject which behaves like a dictionary and it will automatically and
transparently encode data when assigning values. Keys are completely arbitrary and schema is left to the user.
In this case we store themetadata,samplesand all the keys corresponding to datapointsfromthunderpackimportThunderDBwithThunderDB.open('/tmp/thunderpack_test','c')asdb:db['metadata']={'version':'0.1','n_samples':100}keys=[]foriinrange(100):key=f'sample{i:02d}'x=np.random.normal(size=(128,128))y=np.random.normal(size=(128,128))# Thunderpack will serialize the tuple and numpy arrays automaticallydb[key]=(x,y)keys.append(key)db['samples']=keysOnce created, we can read the data usingThunderReader, which as a dict-like APIfromthunderpackimportThunderReaderreader=ThunderReader('/tmp/thunderpack_test')print(reader['metadata'])# {'version': '0.1', 'n_samples': 100}print(reader['samples'][:5])# ['sample00', 'sample01', 'sample02', 'sample03', 'sample04']print(reader['sample00'][0].shape)# (128, 128)Thunderpack provides a PyTorch compatible Dataset object viaThunderDataset, which
assigns a._dbattribute with theThunderReaderobjectfromthunderpack.torchimportThunderDatasetclassMyDataset(ThunderDataset):def__init__(self,file):super().__init__(file)# Access through self._db attributeself.samples=self._db['samples']def__len__(self):returnlen(self.samples)def__getitem__(self,idx):returnself._db[self.samples[idx]]d=MyDataset('/tmp/thunderpack_test')print(len(d))# 100print(d[0][0].shape)# (128, 128)📁 Supported FormatsThunderPack supports a wide range of data formats out of the boxModalitySupported Formats🧮Tensornpy, npz, pt, safetensors📷Imagejpeg, png, bmp, webp🎧Audiowav, flac, ogg, mp3🗂️Tabularcsv, parquet, feather, jsonl📄Documentsjson, yaml, msgpack, txt🗜️Compressionlz4, zstd, gzip, bz2, snappy, brotli🧸Objectpickle↔ Type-Format mappingsThunderPack automatically maps common Python data types to efficient data formatsTypeFormatPIL.ImagePNG or JPEGpandas.DataFrameParquetnp.ndarray,torch.TensorNumpyPack (LZ4)bool,int,float,complex,strMessagePack (LZ4)list,dict,tupleThunderPack (LZ4)✍️ Citation@misc{ortiz2023thunderpack,
    author = {Jose Javier Gonzalez Ortiz},
    title = {The ThunderPack Data Format},
    year = {2023},
    howpublished = {\\url{<https://github.com/JJGO/thunderpack/>}},
}