Trouble with streaming frgfm/imagenette vision dataset with TAR archive
State: closed | Author: frgfm (NONE)
Labels: name: streaming

### Link

https://huggingface.co/datasets/frgfm/imagenette

### Description

Hello there :wave: 

Thanks for the amazing work you've done with HF Datasets! I've just started playing with it, and managed to upload my first dataset. But for the second one, I'm having trouble with the preview since there is some archive extraction involved :sweat_smile: 

Basically, I get a:
```
Status code:   400
Exception:     NotImplementedError
Message:       Extraction protocol for TAR archives like 'https://s3.amazonaws.com/fast-ai-imageclas/imagenette2.tgz' is not implemented in streaming mode. Please use `dl_manager.iter_archive` instead.
```

I've tried several things and checked this issue https://github.com/huggingface/datasets/issues/4181 as well, but no luck so far!

Could you point me in the right direction please? :pray: 

### Owner

Yes

Comments: Hi @frgfm, thanks for reporting.

As the error message says, streaming mode is not supported out of the box when the dataset contains TAR archive files.

To make the dataset streamable, you have to use `dl_manager.iter_archive`.

There are several examples in other datasets, e.g. food101: https://huggingface.co/datasets/food101/blob/main/food101.py

And yes, as the link you pointed out, for the streaming to be possible, the metadata file must be loaded before all of the images:
- either this is the case when iterating the archive (and you get the metadata file before the images)
- or you have to extract the metadata file by hand and upload it separately to the Hub

Hi @albertvillanova :wave:

Thanks! Yeah I saw that but since I didn't have any metadata, I wasn't sure whether I should create them myself.

So one last question:
What is the metadata supposed to be for archives? The relative path of all files in it?
_(Sorry I'm a bit confused since it's quite hard to debug using the single error message from the data preview :sweat_smile: )_

Hi @frgfm, streaming a dataset that contains a TAR file requires some tweaks because (contrary to ZIP files), tha TAR archive does not allow random access to any of the contained member files. Instead they have to be accessed sequentially (in the order in which they were put into the TAR file when created) and yielded.

So when iterating over the TAR file content, when an image file is found, we need to yield it (and not keeping it in memory, which will require huge RAM memory for large datasets). But when yielding an image file, we also need to yield with it what we call "metadata": the class label, and other textual information (for example, for audio files, sometimes we also add info such as the speaker ID, their sex, their age,...).

All this information usually is stored in what we call the metadata file: either a JSON or a CSV/TSV file.

But if this is also inside the TAR archive, we need to find this file in the first place when iterating the TAR archive, so that we already have this information when we find an image file and we can yield the image file and its metadata info.

Therefore:
- either the TAR archive contains the metadata file as the first member when iterating it (something we cannot change as it is done at the creation of the TAR file)
- or if not, then we need to have the metadata file elsewhere
  - in these cases, what we do (if the dataset license allows it) is:
    - we download the TAR file locally, we extract the metadata file and we host the metadata on the Hub
    - we modify the dataset loading script so that it first downloads the metadata file (and reads it) and only then starts iterating the content of the TAR archive file

See an example of this process we recently did for "google/fleurs" (their metadata files for "train" were at the end of the TAR archives, after all audio files): https://huggingface.co/datasets/google/fleurs/discussions/4
- we uploaded the metadata file to the Hub
- we adapted the loading script to use it

Hi @albertvillanova :wave: 

Thanks, since my last message, I went through the repo of https://huggingface.co/datasets/food101/blob/main/food101.py and managed to get it to work in the end :pray: 

Here it is: https://huggingface.co/datasets/frgfm/imagenette

I appreciate you opening an issue to document the process, it might help a few!

Great to see that you manage to make your dataset streamable. :rocket: 

I'm closing this issue, as for the docs update there is another issue opened:
- #4711