Package: tarzan
TarzanTar, as a high performance streamable format, has been widely used in the DL community
(e.g.TorchData,WebDataset).TFDS-like dataset builder API provides a high-level interface for
users to build their own datasets, and is also adopted
byHuggingFace.Why not connect the two? Tarzan provides a minimal high-level API to help users build their own Tar-based datasets. It
also maps well between nested feature and Tar file structure to let you peek into the Tar file without extracting it.InstallationpipinstalltarzanQuick StartDefine your dataset info, which describes the dataset structure and any metadata.fromtarzan.infoimportDatasetInfofromtarzan.featuresimportFeatures,Text,Scalar,Tensor,Audioinfo=DatasetInfo(description="A fake dataset",features=Features({'single':Text(),'nested_list':[Scalar('int32')],'nested_dict':{'inner':Tensor(shape=(None,3),dtype='float32'),},'complex':[{'inner_1':Text(),'inner_2':Audio(sample_rate=16000),}]}),metadata={'version':'1.0.0'})Write your data to Tar files withShardWriter.fromtarzan.writersimportShardWriterwithShardWriter('data_dir',info,max_count=2)aswriter:foriinrange(5):writer.write({'single':'hello','nested_list':[1,2,3],'nested_dict':{'inner':[[1,2,3],[4,5,6]]},'complex':[{'inner_1':'world','inner_2':'audio.wav'}]})The structure of thedata_diris as follows:data_dir
├── 00000.tar
├── 00001.tar
├── 00002.tar
└── dataset_info.jsonmax_countandmax_sizecontrol the maximum number of samples and the maximum size of each shard. Here we set themax_countto 2 to create 3 shards.dataset_info.jsonis a json file serialized from `info, which we rely on to read the data later.catdata_dir/dataset_info.json{"description":"A fake dataset","file_list":["00000.tar","00000.tar","00001.tar","00002.tar"],"features":{"single":{"_type":"Text"},"nested_list":[{"shape":[],"dtype":"int32","_type":"Scalar"}],"nested_dict":{"inner":{"shape":[null,3],"dtype":"float32","_type":"Tensor"}},"complex":[{"inner_1":{"_type":"Text"},"inner_2":{"shape":[null],"dtype":"float32","_type":"Audio","sample_rate":16000}}]},"metadata":{"version":"1.0.0"}}You can peek the tar file without extracting it and it should map well to the nested feature structure.treedata_dir/00000.tar.
├── 0
│   ├── complex
│   │   └── 0
│   │       ├── inner_1
│   │       └── inner_2
│   ├── nested_dict
│   │   └── inner
│   ├── nested_list
│   │   ├── 0
│   │   ├── 1
│   │   └── 2
│   └── single
└── 1
    ├── complex
    │   └── 0
    │       ├── inner_1
    │       └── inner_2
    ├── nested_dict
    │   └── inner
    ├── nested_list
    │   ├── 0
    │   ├── 1
    │   └── 2
    └── single3.Read the dataset withTarReaderfromtarzan.readersimportTarReaderreader=TarReader.from_dataset_info('data_dir/dataset_info.json')fortar_name,idx,exampleinreader:print(tar_name,idx,example)data_dir/00000.tar 0 {'nested_dict': {'inner': array([[1., 2., 3.],
       [4., 5., 6.]], dtype=float32)}, 'single': 'hello', 'complex': [{'inner_1': 'world', 'inner_2': <tarzan.features.audio.AudioDecoder object at 0x7fb8903443d0>}], 'nested_list': [array(1, dtype=int32), array(2, dtype=int32), array(3, dtype=int32)]}
...Note that theAudiofeature is returned as a lazy read objectAudioDecoderto avoid unnecessary read for large audio.