Support zipfiles that span multiple disks?
Author: severo
State: closed
Labels: name: enhancement

name: question

See https://huggingface.co/datasets/PhilEO-community/PhilEO-downstream

The dataset viewer gives the following error:

```
Error code:   ConfigNamesError
Exception:    BadZipFile
Message:      zipfiles that span multiple disks are not supported
Traceback:    Traceback (most recent call last):
                File "/src/services/worker/src/worker/job_runners/dataset/config_names.py", line 67, in compute_config_names_response
                  get_dataset_config_names(
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 347, in get_dataset_config_names
                  dataset_module = dataset_module_factory(
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1871, in dataset_module_factory
                  raise e1 from None
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1846, in dataset_module_factory
                  return HubDatasetModuleFactoryWithoutScript(
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1240, in get_module
                  module_name, default_builder_kwargs = infer_module_for_data_files(
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 584, in infer_module_for_data_files
                  split_modules = {
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 585, in <dictcomp>
                  split: infer_module_for_data_files_list(data_files_list, download_config=download_config)
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 526, in infer_module_for_data_files_list
                  return infer_module_for_data_files_list_in_archives(data_files_list, download_config=download_config)
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 554, in infer_module_for_data_files_list_in_archives
                  for f in xglob(extracted, recursive=True, download_config=download_config)[
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py", line 576, in xglob
                  fs, *_ = fsspec.get_fs_token_paths(urlpath, storage_options=storage_options)
                File "/src/services/worker/.venv/lib/python3.9/site-packages/fsspec/core.py", line 622, in get_fs_token_paths
                  fs = filesystem(protocol, **inkwargs)
                File "/src/services/worker/.venv/lib/python3.9/site-packages/fsspec/registry.py", line 290, in filesystem
                  return cls(**storage_options)
                File "/src/services/worker/.venv/lib/python3.9/site-packages/fsspec/spec.py", line 79, in __call__
                  obj = super().__call__(*args, **kwargs)
                File "/src/services/worker/.venv/lib/python3.9/site-packages/fsspec/implementations/zip.py", line 57, in __init__
                  self.zip = zipfile.ZipFile(
                File "/usr/local/lib/python3.9/zipfile.py", line 1266, in __init__
                  self._RealGetContents()
                File "/usr/local/lib/python3.9/zipfile.py", line 1329, in _RealGetContents
                  endrec = _EndRecData(fp)
                File "/usr/local/lib/python3.9/zipfile.py", line 286, in _EndRecData
                  return _EndRecData64(fpin, -sizeEndCentDir, endrec)
                File "/usr/local/lib/python3.9/zipfile.py", line 232, in _EndRecData64
                  raise BadZipFile("zipfiles that span multiple disks are not supported")
              zipfile.BadZipFile: zipfiles that span multiple disks are not supported
```

The files (https://huggingface.co/datasets/PhilEO-community/PhilEO-downstream/tree/main/data) are:

<img width="629" alt="Capture d’écran 2024-03-11 à 22 07 30" src="https://github.com/huggingface/datasets/assets/1676121/0bb15a51-d54f-4d73-8572-e427ea644b36">


Comments: @severo were you able to solve it?

No. cc @albertvillanova @lhoestq @polinaeterna for an evaluation of what it would take to support this feature.

The underlying issue issue is that the dataset repository has used split ZIP archive files: https://huggingface.co/datasets/PhilEO-community/PhilEO-downstream/tree/main/data
```
downstream_dataset_patches_npzip.z01
downstream_dataset_patches_npzip.z02
...
downstream_dataset_patches_npzip.zip
```
and these are not supported by the Python standard library package `zipfile`.

It's a pretty bad way to share a dataset since one needs to download the full dataset to use it.

We likely won't support this format.

I agree it is a format we maybe should not support: streaming is not possible.

I opened a PR in the reported repo to disable the viewer: https://huggingface.co/datasets/PhilEO-community/PhilEO-downstream/discussions/1