noamaanMulla-03/datasets-issues huggingface dataset:
Support the deserialization of json lines files comprised of lists

### Feature request

I manage a somewhat large and popular Hugging Face dataset known as the [Open Australian Legal Corpus](https://huggingface.co/datasets/umarbutler/open-australian-legal-corpus). I recently updated my corpus to be stored in a json lines file where each line is an array and each element represents a value at a particular column. Previously, my corpus was stored as a json lines file where each line was a dictionary and the keys were the fields.

Essentially, a line in my json lines file used to look like this:
```json
{"version_id":"","type":"","jurisdiction":"","source":"","citation":"","url":"","when_scraped":"","text":""}
```

And now it looks like this:
```json
["","","","","","","",""]
```

This saves 65 bytes per document and allows me very quickly serialise and deserialise documents via `msgspec`.

After making this change, I found that `datasets` was incapable of deserialising my Corpus without a custom loading script, even if I ensured that the `dataset_info` field in my dataset card contained the desired names of my features.

I would like to request that functionality be added to support this format which is more memory-efficent and faster than using dictionaries.

### Motivation

The [documentation](https://huggingface.co/docs/datasets/en/dataset_script) for creating dataset loading scripts asserts that:
> In the next major release, the new safety features of 🤗 Datasets will disable running dataset loading scripts by default, and you will have to pass trust_remote_code=True to load datasets that require running a dataset script.

I would rather not require my users to pass `trust_remote_code=True` which means that I will need built-in support for this format.

### Your contribution

I would be happy to submit a PR for this if this is something you would incorporate into `datasets` and if I can be pointed to where the code would need to go.

Labels: name: enhancement

Update: I ended up deciding to go back to use lines of dictionaries instead of arrays, not because of this issue as my users would be capable of downloading my corpus without `datasets`, but the speed and storage savings are not currently worth breaking my API and harming the backwards compatibility of each new revision.

With that said, for a static dataset that is not regularly updated like mine, and particularly for extremely large datasets with millions or billions of rows, using arrays could have a meaningful impact, and so there is probably still value in supporting this structure, provided the effort is not too much.
Webdataset dataset builder

Allow `load_dataset` to support the Webdataset format.

It allows users to download/stream data from local files or from the Hugging Face Hub.

Moreover it will enable the Dataset Viewer for Webdataset datasets on HF.

## Implementation details

- I added a new Webdataset builder
- dataset with TAR files are now read using the Webdataset builder
- Basic decoding from `webdataset` is used by default, except unsafe ones like pickle
- HF authentication support is done with `xopen`

## TODOS

- [x] tests
- [x] docs


_The documentation is not available anymore as the PR was closed or merged._

I added an error message if the first examples don't appear to be in webdataset format
```
"The TAR archives of the dataset should be in Webdataset format, "
"but the files in the archive don't share the same prefix or the same types."
```

@mariosasko could you review this ? I think it's fine to have webdataset as an optional dependency for now, then depending on usage and user feedbacks see if it makes sense to have our own implementation or not

I just removed the dependency on `webdataset` @mariosasko :)

took your comments into account, lmk if you see anything else
```push_to_hub()``` - Prevent Automatic Generation of Splits 

### Describe the bug

I currently have a dataset which has not been splited. When pushing the dataset to my hugging face dataset repository, it is split into a testing and training set. How can I prevent the split from happening?

### Steps to reproduce the bug

1. Have a unsplit dataset 

```python
Dataset({ features: ['input', 'output', 'Attack', '__index_level_0__'], num_rows: 944685 })
```

2. Push it to huggingface

```python
dataset.push_to_hub(dataset_name)
```

3. On the hugging face dataset repo, the dataset then appears to be splited:

![image](https://github.com/huggingface/datasets/assets/29337128/b4fbc141-42b0-4f49-98df-dd479648fe09)

4. Indeed, when loading the dataset from this repo, the dataset is split in two testing and training set.

```python
from datasets import load_dataset, Dataset

dataset = load_dataset("Jetlime/NF-CSE-CIC-IDS2018-v2", streaming=True)
dataset
```
output: 

```
IterableDatasetDict({
    train: IterableDataset({
        features: ['input', 'output', 'Attack', '__index_level_0__'],
        n_shards: 2
    })
    test: IterableDataset({
        features: ['input', 'output', 'Attack', '__index_level_0__'],
        n_shards: 1
    })
```

### Expected behavior

The dataset shall not be splited, as not requested.

### Environment info

- `datasets` version: 2.19.1
- Platform: Linux-6.2.0-35-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.23.0
- PyArrow version: 15.0.2
- Pandas version: 2.2.2
- `fsspec` version: 2024.3.1
Invalid YAML in README.md: unknown tag !<tag:yaml.org,2002:python/tuple>

### Describe the bug

I wrote a notebook to load an existing dataset, process it, and upload as a private dataset using `dataset.push_to_hub(...)` at the end. The push to hub is failing with:

```
ValueError: Invalid metadata in README.md.
- Invalid YAML in README.md: unknown tag !<tag:yaml.org,2002:python[/tuple](http://192.168.1.128:8888/tuple)> (50:11)

 47 |             - 4
 48 |             - 4
 49 |             - 8
 50 |           - !!binary |
----------------^
 51 |             TwAAAA==
 52 |           '1': !!python[/object/apply](http://192.168.1.128:8888/object/apply):nump ...
```

My dataset has a `train` and `validation` dataset.  These are the features:

```
{'c1': Value(dtype='string', id=None),
 'c2': Value(dtype='string', id=None),
 'c3': [{'value': Value(dtype='string', id=None),
   'start': Value(dtype='int64', id=None),
   'end': Value(dtype='int64', id=None),
   'label': Value(dtype='string', id=None)}],
 'c4': Value(dtype='string', id=None),
 'c5': Value(dtype='string', id=None),
 'c6': Value(dtype='string', id=None),
 'c7': Value(dtype='string', id=None),
 'c8': Sequence(feature=Value(dtype='int32', id=None), length=-1, id=None),
 'c9': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None),
 'c10': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None),
 'labels': Sequence(feature=ClassLabel(names=['O', 'B-ABC', 'I-ABC', ...], id=None), length=-1, id=None),
 'c12': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}
```

This used to work until I decided to cast the `labels` feature to a `Sequence(ClassLabel(...))` type with:

```
ds['train'] = ds['train'].cast_column("labels", Sequence(ClassLabel(names=list(labels))))
ds['validation'] = ds['validation'].cast_column("labels", Sequence(ClassLabel(names=list(labels))))
```

### Steps to reproduce the bug

1. Start with any token classification dataset.
2. Add a `labels` column with data such as `[0,0,0,12,13,13,13,0,0]`.
3. Cast the label column from `Sequence` to `Sequence(ClassLabel))` with:

```
labels = ['O', 'B-TEST', 'I-TEST']
ds = ds.cast_column("labels", Sequence(ClassLabel(names=labels)))
```

4. Push to hub with `ds.push_to_hub("me/awesome-stuff-dataset")`

### Expected behavior

I expected `push_to_hub` to successfully push my dataset to the hub without error.

### Environment info

Python 3.11.9

datasets==2.19.1
transformers==4.41.1
PyYAML==6.0.1
Export Parquet Tablet Audio-Set is null bytes in Arrow 

### Describe the bug

Exporting the processed audio inside the table with the dataset.to_parquet function, the object pyarrow {bytes: null, path: "Some/Path"}
At the same time, the same dataset uploaded to the hub has bit arrays

![Screenshot from 2024-05-27 19-14-49](https://github.com/huggingface/datasets/assets/140120605/ddfba089-426f-4659-9df4-7a634c948b9e)
![Screenshot from 2024-05-27 19-12-51](https://github.com/huggingface/datasets/assets/140120605/4cf8c0a1-650e-491b-86c8-b475c284a021)


### Steps to reproduce the bug

1.Get dataset from audio and cast it 
2.Export and push dataset
3.It’s scary to be indignant at the difference in the uploaded dataset and the fact that it was saved locally
```py
from datasets import Dataset, Audio 
df = Dataset.from_csv("./datasets.csv")
df = df.cast_column("audio", Audio(16000))
df.to_parquet("./datasets.parquet")
df.push_to_hub(repo_id="************", token="**********************")
```

You can use "try replicate case" for this
[replicate_packet.zip](https://github.com/huggingface/datasets/files/15457114/replicate_packet.zip)


### Expected behavior

Two parquet tables identical in content. It is obvious?

### Environment info

Python 3.11+ (I try did it in 3.12 and got same result )
use_custom_baseline still produces errors for bertscore

`metric = load_metric('bertscore')`
`a1 = "random sentences"`
`b1 = "random sentences"`
`metric.compute(predictions = [a1], references = [b1], lang = 'en')`

`Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/stephen_chan/.local/lib/python3.6/site-packages/datasets/metric.py", line 393, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/stephen_chan/.cache/huggingface/modules/datasets_modules/metrics/bertscore/361e597a01a41d6cf95d94bbfb01dea16261687abc0c6c74cc9930f80488f363/bertscore.py", line 108, in _compute
    hashcode = bert_score.utils.get_hash(model_type, num_layers, idf, rescale_with_baseline)
TypeError: get_hash() missing 1 required positional argument: 'use_custom_baseline'`

Adding 'use_custom_baseline = False' as an argument produces this error

`Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/stephen_chan/.local/lib/python3.6/site-packages/datasets/metric.py", line 393, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
TypeError: _compute() got an unexpected keyword argument 'use_custom_baseline'`

This is on Ubuntu 18.04, Python 3.6.9, datasets version 1.1.2

Labels: name: metric bug

Thanks for reporting ! That's a bug indeed
If you want to contribute, feel free to fix this issue and open a PR :)

This error is because of a mismatch between `datasets` and `bert_score`. With `datasets=1.1.2` and `bert_score>=0.3.6` it works ok. So `pip install -U bert_score` should fix the problem. 

Thanks for the heads up @pvl and for the PR as well :)

Hello everyone,

I think the  problem is not solved: 

```
from datasets import load_metric
metric=load_metric('bertscore')
metric.compute(
    predictions=predictions,
    references=references,
    lang='fr',
    rescale_with_baseline=True
)
TypeError: get_hash() missing 2 required positional arguments: 'use_custom_baseline' and 'use_fast_tokenizer'
```
This code is produced using `Python 3.6.9 datasets==1.1.2 and bert_score==0.3.10`

Hi ! This has been fixed by https://github.com/huggingface/datasets/pull/2770, we'll do a new release soon to make the fix available :)

In the meantime please use an older version of `bert_score`
Column order is nondeterministic when loading from JSON

As reported by @meg-huggingface, the order of the JSON object keys is not preserved while loading a dataset from a JSON file with a list of objects.

For example, when loading a JSON files with a list of objects, each with the following ordered keys:
- [ID, Language, Topic], 

the resulting dataset may have columns:
- [ID, Topic, Language], or
- [Topic, Language, ID], or
- [Topic, ID, Language],...

This issue is caused by the use of a Python set (which does not preserve the order):
https://github.com/huggingface/datasets/blob/60d21efbc01e15d0b596ac1072750cbecd91548a/src/datasets/packaged_modules/json/json.py#L168
introduced in
- #5772

Labels: name: bug
Avoid downloading the whole dataset when only README.me has been touched on hub.

### Feature request

`datasets.load_dataset()` triggers a new download of the **whole dataset** when the README.md file has been touched on huggingface hub, even if data files / parquet files are the exact same.

I think the current behaviour of the load_dataset function is triggered whenever a change of the hash of latest commit on huggingface hub, but  is there a clever way to only download again the dataset **if and only if** data is modified ?  

### Motivation

The current behaviour is a waste of network bandwidth / disk space / research time.

### Your contribution

I don't have time to submit a PR, but I hope a simple solution will emerge from this issue ! 

Labels: name: enhancement

you're right, we're tackling this here: https://github.com/huggingface/dataset-viewer/issues/2757

@severo : great !
[2.12.0] DatasetDict.save_to_disk not saving to S3

### Describe the bug

When trying to save a `DatasetDict` to a private S3 bucket using `save_to_disk`, the artifacts are instead saved locally, and not in the S3 bucket.

I have tried using the deprecated `fs` as well as the `storage_options` arguments and I get the same results.

### Steps to reproduce the bug

1. Create a DatsetDict `dataset`
2. Create a S3FileSystem object
`s3 = datasets.filesystems.S3FileSystem(key=aws_access_key_id, secret=aws_secret_access_key)`
3. Save using `dataset_dict.save_to_disk(f"{s3_bucket}/{s3_dir}/{dataset_name}", storage_options=s3.storage_options)` or `dataset_dict.save_to_disk(f"{s3_bucket}/{s3_dir}/{dataset_name}", fs=s3)`
4. Check the corresponding S3 bucket and verify nothing has been uploaded
5. Check the path at f"{s3_bucket}/{s3_dir}/{dataset_name}" and verify that files have been saved there

### Expected behavior

Artifacts are uploaded at the f"{s3_bucket}/{s3_dir}/{dataset_name}" S3 location.

### Environment info

- `datasets` version: 2.12.0
- Platform: macOS-13.3.1-x86_64-i386-64bit
- Python version: 3.11.2
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.1


Hi ! Can you try adding the `s3://` prefix ?
```python
f"s3://{s3_bucket}/{s3_dir}/{dataset_name}"
```

Ugh, yeah that was it. Thank you!

Hi @thejamesmarq, by any chance, did you use multiprocessing `num_proc > 1` when saving your dataset on the s3 bucket ? I'm struggling making it work in a multiprocessing setting while everything works fine with one processor.
Fix expected splits when passing data_files or dir

reported on slack:

The following code snippet gives an error with v2.19 but not with v2.18:
from datasets import load_dataset
```
dataset = load_dataset(
    "lvwerra/stack-exchange-paired",
    split="train",
    cache_dir=None,
    data_dir="data/rl",
)
```
and the error is:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 2609, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1027, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1140, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/usr/local/lib/python3.10/dist-packages/datasets/utils/info_utils.py", line 92, in verify_splits
    raise ExpectedMoreSplits(str(set(expected_splits) - set(recorded_splits)))
datasets.utils.info_utils.ExpectedMoreSplits: {'test'}
```



The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6938). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

fix is included in https://github.com/huggingface/datasets/pull/6925
ExpectedMoreSplits error when using data_dir

As reported by @regisss, an `ExpectedMoreSplits` error is raised when passing `data_dir`:
```python
from datasets import load_dataset

dataset = load_dataset(
    "lvwerra/stack-exchange-paired",
    split="train",
    cache_dir=None,
    data_dir="data/rl",
)
```
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 2609, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1027, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1140, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/usr/local/lib/python3.10/dist-packages/datasets/utils/info_utils.py", line 92, in verify_splits
    raise ExpectedMoreSplits(str(set(expected_splits) - set(recorded_splits)))
datasets.utils.info_utils.ExpectedMoreSplits: {'test'}
```

Labels: name: bug
Enable Sharding to Equal Sized Shards

### Feature request

Add an option when sharding a dataset to have all shards the same size. Will be good to provide both an option of duplication, and by truncation.

### Motivation

Currently the behavior of sharding is "If n % i == l, then the first l shards will have length (n // i) + 1, and the remaining shards will have length (n // i).". However, when using FSDP we want the shards to have the same size. This requires the user to manually handle this situation, but it will be nice if we had an option to shard the dataset into equally sized shards. 

### Your contribution

For now just a PR. I can also add code that does what is needed, but probably not efficient.
Shard to equal size by duplication:
```
remainder = len(dataset) % num_shards
num_missing_examples = num_shards - remainder
duplicated = dataset.select(list(range(num_missing_examples)))
dataset = concatenate_datasets([dataset, duplicated])
shard = dataset.shard(num_shards, shard_idx)
```
Or by truncation:
```
shard = dataset.shard(num_shards, shard_idx)
num_examples_per_shard = len(dataset) // num_shards
shard = shard.select(list(range(num_examples_per_shard)))
```

Labels: name: enhancement
to_tf_dataset: Visible devices cannot be modified after being initialized

### Describe the bug

When trying to use to_tf_dataset with a custom data_loader collate_fn when I use parallelism I am met with the following error as many times as number of workers there were in ``num_workers``.

  File "/opt/miniconda/envs/env/lib/python3.11/site-packages/multiprocess/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/miniconda/envs/env/lib/python3.11/site-packages/multiprocess/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/miniconda/envs/env/lib/python3.11/site-packages/datasets/utils/tf_utils.py", line 438, in worker_loop
    tf.config.set_visible_devices([], "GPU")  # Make sure workers don't try to allocate GPU memory
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda/envs/env/lib/python3.11/site-packages/tensorflow/python/framework/config.py", line 566, in set_visible_devices
    context.context().set_visible_devices(devices, device_type)
  File "/opt/miniconda/envs/env/lib/python3.11/site-packages/tensorflow/python/eager/context.py", line 1737, in set_visible_devices
    raise RuntimeError(
RuntimeError: Visible devices cannot be modified after being initialized

### Steps to reproduce the bug

1. Download a dataset using HuggingFace load_dataset
2. Define a function that transforms the data in some way to be used in the collate_fn argument
3. Provide a ``batch_size`` and ``num_workers`` value in the ``to_tf_dataset`` function
4. Either retrieve directly or use tfds benchmark to test the dataset

``` python
from datasets import load_datasets
import tensorflow_datasets as tfds
from keras_cv.layers import Resizing

def data_loader(examples):
      x = Resizing(examples[0]['image'], 256, 256, crop_to_aspect_ratio=True)
      return {X[0]: x}

ds = load_datasets("logasja/FDF", split="test")

ds = ds.to_tf_dataset(collate_fn=data_loader, batch_size=16, num_workers=2)

tfds.benchmark(ds)
```

### Expected behavior

Use multiple processes to apply transformations from the collate_fn to the tf dataset on the CPU.

### Environment info

- `datasets` version: 2.19.1
- Platform: Linux-6.5.0-1023-oracle-x86_64-with-glibc2.35
- Python version: 3.11.8
- `huggingface_hub` version: 0.22.2
- PyArrow version: 15.0.2
- Pandas version: 2.2.1
- `fsspec` version: 2024.2.0
imagenet-1k downloads over and over

### Describe the bug

What could be causing this?
```
$ python3
Python 3.8.13 (default, Mar 28 2022, 11:38:47) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from datasets import load_dataset
>>> load_dataset("imagenet-1k")
Downloading builder script: 100%|██████████| 4.72k/4.72k [00:00<00:00, 7.51MB/s]
Downloading readme: 100%|███████████████████| 85.4k/85.4k [00:00<00:00, 510kB/s]
Downloading extra modules: 100%|████████████| 46.4k/46.4k [00:00<00:00, 300kB/s]
Downloading data: 100%|████████████████████| 29.1G/29.1G [19:36<00:00, 24.8MB/s]
Downloading data: 100%|████████████████████| 29.3G/29.3G [08:38<00:00, 56.5MB/s]
Downloading data: 100%|████████████████████| 29.0G/29.0G [09:26<00:00, 51.2MB/s]
Downloading data: 100%|████████████████████| 29.2G/29.2G [09:38<00:00, 50.6MB/s]
Downloading data: 100%|███████████████████▉| 29.2G/29.2G [09:37<00:00, 44.1MB/s^Downloading data:   0%|                     | 106M/29.1G [00:05<23:49, 20.3MB/s]
```

### Steps to reproduce the bug

See above commands/code

### Expected behavior

imagenet-1k is downloaded

### Environment info

- `datasets` version: 2.14.6
- Platform: Linux-6.2.0-34-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- Huggingface_hub version: 0.15.1
- PyArrow version: 14.0.0
- Pandas version: 1.5.2


Have you solved this problem?