WebDataset with different prefixes are unsupported
Opened by hlky (closed)

### Describe the bug

Consider a WebDataset with multiple images for each item where the number of images may vary: [example](https://huggingface.co/datasets/bigdata-pw/fashion-150k)

Due to this [code](https://github.com/huggingface/datasets/blob/87f4c2088854ff33e817e724e75179e9975c1b02/src/datasets/packaged_modules/webdataset/webdataset.py#L76-L80) an error is given.
```
The TAR archives of the dataset should be in WebDataset format, but the files in the archive don't share the same prefix or the same types.
```

The purpose of this check is unclear because PyArrow supports different keys.

Removing the check allows the dataset to be loaded and there's no issue when iterating through the dataset.

```
>>> from datasets import load_dataset
>>> path = "shards/*.tar"
>>> dataset = load_dataset("webdataset", data_files={"train": path}, split="train", streaming=True)
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 152/152 [00:00<00:00, 56458.93it/s]
>>> dataset
IterableDataset({
    features: ['__key__', '__url__', '1.jpg', '2.jpg', '3.jpg', '4.jpg', 'json'],
    n_shards: 152
})
```

### Steps to reproduce the bug

```python
from datasets import load_dataset
load_dataset("bigdata-pw/fashion-150k")
```

### Expected behavior

Dataset loads without error

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-5.14.0-467.el9.x86_64-x86_64-with-glibc2.34
- Python version: 3.9.19
- `huggingface_hub` version: 0.23.4
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0

Comments:
Since `datasets` uses is built on Arrow to store the data, it requires each sample to have the same columns.

This can be fixed by specifyign in advance the name of all the possible columns in the `dataset_info` in YAML, and missing values will be `None`

Thanks. This currently doesn't work for WebDataset because there's no `BuilderConfig` with `features` and in turn `_info` is missing `features=self.config.features`. I'll prepare a PR to fix this.

Note it may be useful to add the [expected format of `features`](https://github.com/huggingface/datasets/blob/16fa4421f44b22bbbc607f379a93f45af468d1fc/src/datasets/features/features.py#L1757) to the documentation for [`Builder Parameters`](https://huggingface.co/docs/datasets/repository_structure#builder-parameters).


Oh good catch ! thanks

> Note it may be useful to add the [expected format of features](https://github.com/huggingface/datasets/blob/16fa4421f44b22bbbc607f379a93f45af468d1fc/src/datasets/features/features.py#L1757) to the documentation for [Buil](https://huggingface.co/docs/datasets/repository_structure#builder-parameters)

Good idea, let me open a PR

#7060 

Actually I just tried with `datasets` on the `main` branch and having `features` defined in `dataset_info` worked for me

```python
>>> list(load_dataset("/Users/quentinlhoest/tmp", streaming=True, split="train"))
[{'txt': 'hello there\n', 'other': None}]
```
where `tmp` contains data.tar with "hello there\n" in a text file and the README.md:
```
---
dataset_info:
  features:
  - name: txt
    dtype: string
  - name: other
    dtype: string
---

This is a dataset card
```

What error did you get when you tried to specify the columns in `dataset_info` ?

If you review the changes in #7060 you'll note that `features` are not passed to `DatasetInfo`.

In your case the features are being extracted by [this code](https://github.com/huggingface/datasets/blob/e83d6fa574710fcb44e341087239d2687183f62b/src/datasets/packaged_modules/webdataset/webdataset.py#L72-L98).

Try with the `Steps to reproduce the bug`. It's the same error mentioned in `Describe the bug` because `features` are not passed to `DatasetInfo`.

`features` are [not used](https://github.com/huggingface/datasets/blob/e83d6fa574710fcb44e341087239d2687183f62b/src/datasets/builder.py#L365-L366) when the `BuilderConfig` has no `features` attribute. `WebDataset` uses the default [`BuilderConfig`](https://github.com/huggingface/datasets/blob/e83d6fa574710fcb44e341087239d2687183f62b/src/datasets/builder.py#L101-L124).

There is a [warning](https://github.com/huggingface/datasets/blob/e83d6fa574710fcb44e341087239d2687183f62b/src/datasets/load.py#L640-L648) that `features` are ignored.

Note that as mentioned in `Describe the bug` this could also be resolved by removing the check [here](https://github.com/huggingface/datasets/blob/e83d6fa574710fcb44e341087239d2687183f62b/src/datasets/packaged_modules/webdataset/webdataset.py#L76-L80) because Arrow actually handles this itself, Arrow sets any missing fields to `None`, at least in my case.

Note for anyone else who encounters this issue, every dataset type except folder-based types supported features in the [documented](https://huggingface.co/docs/datasets/repository_structure#builder-parameters) manner; [Arrow](https://github.com/huggingface/datasets/blob/e83d6fa574710fcb44e341087239d2687183f62b/src/datasets/packaged_modules/arrow/arrow.py#L15-L21), [csv](https://github.com/huggingface/datasets/blob/e83d6fa574710fcb44e341087239d2687183f62b/src/datasets/packaged_modules/csv/csv.py#L25-L68), [generator](https://github.com/huggingface/datasets/blob/e83d6fa574710fcb44e341087239d2687183f62b/src/datasets/packaged_modules/generator/generator.py#L8-L19), [json](https://github.com/huggingface/datasets/blob/e83d6fa574710fcb44e341087239d2687183f62b/src/datasets/packaged_modules/json/json.py#L42-L52), [pandas](https://github.com/huggingface/datasets/blob/e83d6fa574710fcb44e341087239d2687183f62b/src/datasets/packaged_modules/pandas/pandas.py#L14-L20), [parquet](https://github.com/huggingface/datasets/blob/e83d6fa574710fcb44e341087239d2687183f62b/src/datasets/packaged_modules/parquet/parquet.py#L16-L24), [spark](https://github.com/huggingface/datasets/blob/e83d6fa574710fcb44e341087239d2687183f62b/src/datasets/packaged_modules/spark/spark.py#L31-L37), [sql](https://github.com/huggingface/datasets/blob/e83d6fa574710fcb44e341087239d2687183f62b/src/datasets/packaged_modules/sql/sql.py#L24-L35) and [text](https://github.com/huggingface/datasets/blob/e83d6fa574710fcb44e341087239d2687183f62b/src/datasets/packaged_modules/text/text.py#L18-L27). `WebDataset` is different and requires [`dataset_info` which is vaguely documented](https://huggingface.co/docs/datasets/dataset_script#optional-generate-dataset-metadata) under dataset loading scripts.

Thanks for explaining. I see the Dataset Viewer is still failing - I'll update `datasets` in the Viewer to fix this