.load_dataset() method defaults to zstandard
Opened by ElleLeonne (closed)

### Describe the bug

Regardless of what method I use, datasets defaults to zstandard for unpacking my datasets.

This is poor behavior, because not only is zstandard not a dependency in the huggingface package (and therefore, your dataset loading will be interrupted while it asks you to install the package), but it happens on datasets that are uploaded in json format too, meaning the dataset loader will attempt to convert the data to a zstandard compatible format, and THEN try to unpackage it.

My 4tb drive runs out of room when using zstandard on slimpajama. It loads fine on 1.5tb when using json, however I lack the understanding of the "magic numbers" system used to select the unpackaging algorithm, so I can't push a change myself.

Commenting out this line, in "/datasets/utils/extract.py" fixes the issue, and causes SlimPajama to properly extract using rational amounts of storage, however it completely disables zstandard, which is probably undesirable behavior. Someone with an understanding of the "magic numbers" system should probably take a pass over this issue.

```
class Extractor:
    #  Put zip file to the last, b/c it is possible wrongly detected as zip (I guess it means: as tar or gzip)
    extractors: Dict[str, Type[BaseExtractor]] = {
        "tar": TarExtractor,
        "gzip": GzipExtractor,
        "zip": ZipExtractor,
        "xz": XzExtractor,
        #"zstd": ZstdExtractor,   # This line needs to go, in order for datasets to work w/o non-dependent packages
        "rar": RarExtractor,
        "bz2": Bzip2Extractor,
        "7z": SevenZipExtractor,  # <Added version="2.4.0"/>
        "lz4": Lz4Extractor,  # <Added version="2.4.0"/>
    }
```

### Steps to reproduce the bug

'''
from datasaets import load_dataset
load_dataset(path="/cerebras/SlimPajama-627B")
'''

This alone should trigger the error on any system that does not have zstandard pip installed.

### Expected behavior

This repository (which is encoded in json format, not zstandard) should check whether zstandard is installed before defaulting to it. Additionally, using zstandard should not use more than 3x the required space that other extraction mechanisms use.

### Environment info

- `datasets` version: 2.17.1
- Platform: Linux-6.5.0-18-generic-x86_64-with-glibc2.35
- Python version: 3.12.0
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0

Comments:
The dataset is made of JSON files compressed using zstandard, as you can see here: https://huggingface.co/datasets/cerebras/SlimPajama-627B/tree/main/test/chunk1

That's why it asks for zstandard to be installed.

Though I'm intrigued that you manage to load the dataset without zstandard installed. Maybe `pyarrow` that we use to load JSON data under the hood got support for zstandard at one point.

> The dataset is made of JSON files compressed using zstandard, as you can see here: https://huggingface.co/datasets/cerebras/SlimPajama-627B/tree/main/test/chunk1
> 
> That's why it asks for zstandard to be installed.
> 
> Though I'm intrigued that you manage to load the dataset without zstandard installed. Maybe `pyarrow` that we use to load JSON data under the hood got support for zstandard at one point.

Question, then.

When I loaded this dataset back in October, it downloaded all the files, and then loaded into memory just fine.

NOW, it has to sit there and unpack all these zstd files (3.6TB worth). Further, when they're in my harddrive, they're regular json files. It's only when looking at the LFS, or when the loading script runs, that I get asked to install zstd.

My question is, **is this normal?** As far as I can tell, there's no reason the dataset or the loading methods should have changed between then and now. Was my old behavior flawed, and the new behavior correct?

I mean, I got it working eventually, but it was pulling teeth, and it still doesn't load right, as I had to unpack each chunk separately, so there's no clean mapping between the chunks and the broader dataset.

The `ZstdExtractor` has been added 3 years ago and we haven't touched it since then. Same for the JSON loader.

`zstandard` is required as soon as you try to load a file with the `.zstd` extension or if a file starts with the Zstandard magic number `b"\x28\xb5\x2f\xfd"` (used to recognize Zstandard files).

Note that the extraction only has to happen once - if you reload the dataset it will be reloaded from your cache directly.

Not sure what happened between October and now unfortunately

Understood, thank you for clarifying that for me.

I'll look into how best to collate my stack of batches w/o creating duplicate arrow tables for each one.