Support for multiple configs in packaged modules via metadata yaml info
State: closed

will solve https://github.com/huggingface/datasets/issues/5209 and https://github.com/huggingface/datasets/issues/5151 and many other...

Config parameters for packaged builders are parsed from `“builder_config”` field in README.md file (separate firs-level field, not part of “dataset_info”), example:
```yaml
---
dataset_info:
...
configs:
  - config_name: v1
    data_dir: v1
    drop_labels: true
  - config_name: v2
    data_dir: v2
    drop_labels: false
```

I tried to align packaged builders with custom configs parsed from metadata with scripts dataset builder as much as possible. Their builders are created dynamically (see `configure_builder_class()` in load.py`) and have `BUILDER_CONFIGS` attribute filled with `BuilderConfig` objects in the same way as for datasets with script.

## load_dataset
1. If there is single config in meta and it doesn’t have a name, the name becomes “default” (as we do for “dataset_info”), [example](https://huggingface.co/datasets/polinaeterna/audiofolder_one_default_config_in_metadata/blob/main/README.md):
```python
load_dataset("ds") == load_dataset("ds", "default")   # load with the params provided in metadata

load_dataset("ds", "random name")  # ValueError: BuilderConfig 'random_name' not found. Available: ['default']
```
2. If there is single config in metadata with `config_name` provided, it becomes a default one (loaded when no `config_name` is specified, [example](https://huggingface.co/datasets/polinaeterna/audiofolder_one_nondefault_config_in_metadata)
```python

load_dataset("ds") == load_dataset("ds", "custom")   # load with the params provided in meta

load_dataset("ds", "random name")  # ValueError: BuilderConfig 'random_name' not found. Available: ['custom']
```
3. If there are several configs in metadata with names [example](https://huggingface.co/datasets/polinaeterna/audiofolder_two_configs_in_metadata/blob/main/README.md)
```python
load_dataset("ds", "v1")  # load with "v1" params

load_dataset("ds", "v2")  # load with "v2" params

load_dataset("ds")  # ValueError: BuilderConfig 'default' not found. Available: ['v1', 'v2']

```
Thanks to @lhoestq and [this change](https://github.com/polinaeterna/datasets/pull/1), it's possible to add `"default"` field in yaml and set it to True, to make the config a default one (loaded when no config is specified):
```yaml
configs:
  - config_name: v1
    drop_labels: true
    default: true
  - config_name: v2
...
```
then `load_dataset("ds") == load_dataset("ds", "v1")`.

## dataset_name and cache
I decided that it’s reasonable to add a `dataset_name` attribute to `DatasetBuilder` class which would be equal to `name` for scripts dataset but reflect a real dataset name for packaged builders (last part of path/name from hub). This is mostly to reorganize cache structure (I believe we can do this in the major release?) because otherwise, with custom configs for packaged builders which were all stored in the same directory, i it was becoming a mess. And in general it makes much more sense like this, from datasets server perspective too, though it’s a breaking change 

So the cache dir has the following structure: `"{namespace__}<dataset_name>/<config_name>/<version>/<hash>/"` and arrow/parquet filenames are also `"<dataset_name>-<split>.arrow"`. 
For example `polinaeterna___audiofolder_two_configs_in_metadata/v1-5532fac9443ea252/0.0.0/6cbdd16f8688354c63b4e2a36e1585d05de285023ee6443ffd71c4182055c0fc/` for `polinaeterna/audiofolder_two_configs_in_metadata` Hub dataset, train arrow file is `audiofolder_two_configs_in_metadata-train.arrow`.

For script datasets it remains unchanged.

## push_to_hub
To support custom configs with `push_to_hub`, the data is put under directory named either as `<config_name>` if `config_name` is **not** "default" or "data" if `config_name` is omitted or "default" (for backward compatibility). `"builder_config"` field is added to README.md, with `config_name` (optional) and `data_files` fields. for `"data_files"`, `"pattern"` parameter is introduced, to resolve data files correctly, see https://github.com/polinaeterna/datasets/pull/1.
 - `ds.push_to_hub("ds")` --> one config ("default"), put under "data" directory, [example](https://huggingface.co/datasets/polinaeterna/push_to_hub_single_config/blob/main/README.md)
```yaml
dataset_info:
...
configs:
  data_files:
  - split: train
    pattern: data/train-*
...
``` 
- `ds.push_to_hub("ds", "custom")` --> put under "custom" directory, [example](https://huggingface.co/datasets/polinaeterna/push_to_hub_singe_nondefault_config/blob/main/README.md)

```yaml
configs:
  config_name: custom
  data_files:
    - split: train
      path: custom/train-*
...
``` 

 - for many configs, [example](https://huggingface.co/datasets/polinaeterna/push_to_hub_many_configs/blob/main/README.md):
```yaml
configs:
  - config_name: v1
    data_files:
      - split: train
        path: v1/train-*
...
  - config_name: v2
    data_files:
      - split: train
        path: v2/train-*
...
```
Thanks to @lhoestq and https://github.com/polinaeterna/datasets/pull/1, when pushing to datasets created **before** this change, README.md is updated accordingly (config for old data is added along with the one that is being pushed).  

`"dataset_info"` yaml field is updated accordingly (new configs are added).
This shouldn't break anything!

TODO in separate PRs:
- [x] docs
- [ ] probably update test cli util (make --save_info not rewrite `builder_config` in readme)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

feel free to merge `main` into your PR to fix the CI :)

Let me see if I can fix the pattern thing ^^'

Hmm I think it would be easier to specify the `data_files` in the end, because having a split pattern like `{split}-...` at the root of the repository can lead to unexpected behaviors IMO, and we probably don't want to have a different behavior for `data_files` depending if it's inside a `data_dir` or not

Maybe something like
```yaml
builder_config:
  data_dir: data_dir
  data_files:
  - split: train
    pattern: train-[0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*
```

 > Also, I'm not sure if it's a good idea to have this field in the YAML metadata - Transformers use this part of the card only for Hub-related stuff (widgets, tags, CO2 emission, etc.), and I think we should aim to do the same in Datasets. We could achieve this by having these kwargs in a special file (they can be seen as a faster way of defining a builder (builder script) that subclasses a packaged builder) and removing the dataset_info field (the only useful info there seem to be features and we can fetch those directly from a dataset script/Parquet files).

Something like `config.json`?

```json
{
    "data_dir": "data"
    "data_files":  {
        "train": "train-[0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*"
    }
}
```

we could also support lists for several configs

opened https://github.com/huggingface/datasets/issues/5694

I opened a PR to this PR to add data_files in YAML: https://github.com/polinaeterna/datasets/pull/1

```yaml
builder_config:
  data_files:
  - split: train
    pattern: data/train-*
```

Let me open a PR to see if I can move the data files resolution outside of the MetadataConfigs to not modify it in-place

I wonder if we can make the cache backward compatible: we could just check if the cache directory with the old path exists. It will be useful for the research team which has a big datasets cache

> I wonder if we can make the cache backward compatible: we could just check if the cache directory with the old path exists. It will be useful for the research team which has a big datasets cache

![image](https://github.com/huggingface/datasets/assets/16348744/90a96e79-2a0d-4d37-95bd-b75fa962c094)

In the next PR maybe? :D 
It's possible but requires some additional logic to correctly pass old `config_kwargs` (which used to include `data_files` but now it's `None` for builders from metadata) to generate the hash which is used to create the path.

If we only consider datasets that were pushed to hub, it's just a matter of using `"{username}__parquet"` instead of `"{username}__{dataset_name}"` in the cache directory name. The hashes stay the same :)

EDIT: and the config name
EDIT2: and the arrow file names

Did a small PR for backward compatibility, it was easy to add in the end: https://github.com/polinaeterna/datasets/pull/3

Just created a branch [dev-3.0](https://github.com/huggingface/datasets/tree/dev-3.0) in which we can merge this one and the other datasets 3.0 related PRs

@lhoestq why can't we merge it in main?

We can, it was just in case we had other things to merge after @mariosasko or @albertvillanova 's reviews

@lhoestq @albertvillanova @mariosasko we agreed on having `configs` (in plural) as a metadata field in readme but apparently Hub's yaml validation doesn't allow it to be not a list :D 
![image](https://github.com/huggingface/datasets/assets/16348744/52131ee8-80e0-4f6e-90cd-8ff83caf4625)
(with `config` (in singular) it works)

edit: and now the tests for hub datasets with metadata configs are failing because I cannot change the yaml there...

> we agreed on having configs (in plural) as a metadata field in readme but apparently Hub's yaml validation doesn't allow it to be not a list :D

If the `configs` field is specified in the YAML, the Hub can use it to [improve](https://github.com/huggingface/moon-landing/blob/97aca4cac32fbb7d84ce5eba9b18afad87968c4a/server/views/components/DatasetLibraryModal/datasetLibrarySnippets.ts#L11) the `Use in dataset library` snippet by listing the possible config values in `load_dataset`. So I think this needs to be fixed on the Hub side.

PS: I couldn't find an instance of someone using this field on the Hub, so I think using it for this feature is OK.

> I couldn't find an instance of someone using this field on the Hub, so I think using it for this feature is OK.

@mariosasko I think it's because @lhoestq renamed `configs` to `config_names` in all canonical datasets :D so yes, `configs` field is now supposed to include custom configuration parameters introduced in this PR, and `config_names` is used (not really used lol) for list of strings of config names. It's being fixed on the Hub's side https://github.com/huggingface/moon-landing/pull/6490

after more thought I agree it's maybe overkill to do a major release for this one, since we have a good backward compatibility

There is one edge case I forgot to mention in the reviews - I think it's a good idea to support passing config params that are functions (Pandas uses them a lot) using this API (e.g. `converters` in the CSV config for converting a string column into a sequence). I see two solutions: string blocks with Python code in YAML or PyYAML [tags](https://pyyaml.org/wiki/PyYAMLDocumentation#yaml-tags-and-python-types). 

But I think this can be addressed later.

I'm resolving the conflicts and writing some docs :) let's merge this soon !

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005868 / 0.011353 (-0.005485) | 0.003544 / 0.011008 (-0.007464) | 0.080329 / 0.038508 (0.041821) | 0.061072 / 0.023109 (0.037963) | 0.307802 / 0.275898 (0.031904) | 0.340353 / 0.323480 (0.016873) | 0.004665 / 0.007986 (-0.003321) | 0.002779 / 0.004328 (-0.001550) | 0.062065 / 0.004250 (0.057815) | 0.046350 / 0.037052 (0.009297) | 0.312045 / 0.258489 (0.053556) | 0.353524 / 0.293841 (0.059683) | 0.026965 / 0.128546 (-0.101581) | 0.007906 / 0.075646 (-0.067740) | 0.260678 / 0.419271 (-0.158593) | 0.044167 / 0.043533 (0.000634) | 0.309757 / 0.255139 (0.054618) | 0.340188 / 0.283200 (0.056988) | 0.020440 / 0.141683 (-0.121243) | 1.486886 / 1.452155 (0.034732) | 1.548330 / 1.492716 (0.055614) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.188658 / 0.018006 (0.170652) | 0.422204 / 0.000490 (0.421715) | 0.003508 / 0.000200 (0.003308) | 0.000068 / 0.000054 (0.000013) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025173 / 0.037411 (-0.012238) | 0.072868 / 0.014526 (0.058343) | 0.084817 / 0.176557 (-0.091739) | 0.151667 / 0.737135 (-0.585468) | 0.085632 / 0.296338 (-0.210706) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.400998 / 0.215209 (0.185789) | 4.022274 / 2.077655 (1.944619) | 2.025768 / 1.504120 (0.521648) | 1.874193 / 1.541195 (0.332998) | 2.006537 / 1.468490 (0.538047) | 0.501799 / 4.584777 (-4.082978) | 2.987487 / 3.745712 (-0.758225) | 4.552295 / 5.269862 (-0.717566) | 2.775859 / 4.565676 (-1.789817) | 0.057596 / 0.424275 (-0.366679) | 0.006449 / 0.007607 (-0.001158) | 0.470776 / 0.226044 (0.244732) | 4.725933 / 2.268929 (2.457005) | 2.480130 / 55.444624 (-52.964494) | 2.183919 / 6.876477 (-4.692558) | 2.408052 / 2.142072 (0.265979) | 0.584038 / 4.805227 (-4.221190) | 0.124964 / 6.500664 (-6.375701) | 0.060939 / 0.075469 (-0.014530) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.221263 / 1.841788 (-0.620524) | 18.326372 / 8.074308 (10.252064) | 13.398937 / 10.191392 (3.207545) | 0.149153 / 0.680424 (-0.531271) | 0.016941 / 0.534201 (-0.517260) | 0.332106 / 0.579283 (-0.247177) | 0.339958 / 0.434364 (-0.094406) | 0.378125 / 0.540337 (-0.162212) | 0.517787 / 1.386936 (-0.869149) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005927 / 0.011353 (-0.005426) | 0.003607 / 0.011008 (-0.007402) | 0.062925 / 0.038508 (0.024417) | 0.058676 / 0.023109 (0.035566) | 0.362129 / 0.275898 (0.086231) | 0.395864 / 0.323480 (0.072384) | 0.004652 / 0.007986 (-0.003334) | 0.002893 / 0.004328 (-0.001435) | 0.062696 / 0.004250 (0.058445) | 0.049988 / 0.037052 (0.012935) | 0.365366 / 0.258489 (0.106877) | 0.412326 / 0.293841 (0.118485) | 0.027118 / 0.128546 (-0.101429) | 0.008179 / 0.075646 (-0.067467) | 0.068048 / 0.419271 (-0.351223) | 0.041065 / 0.043533 (-0.002468) | 0.359858 / 0.255139 (0.104719) | 0.386589 / 0.283200 (0.103390) | 0.020467 / 0.141683 (-0.121216) | 1.438070 / 1.452155 (-0.014084) | 1.479617 / 1.492716 (-0.013099) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.231516 / 0.018006 (0.213510) | 0.413407 / 0.000490 (0.412917) | 0.000358 / 0.000200 (0.000158) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026071 / 0.037411 (-0.011340) | 0.076486 / 0.014526 (0.061960) | 0.085943 / 0.176557 (-0.090613) | 0.138087 / 0.737135 (-0.599048) | 0.087466 / 0.296338 (-0.208872) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.417711 / 0.215209 (0.202502) | 4.171915 / 2.077655 (2.094260) | 2.140677 / 1.504120 (0.636557) | 1.960164 / 1.541195 (0.418969) | 2.002134 / 1.468490 (0.533644) | 0.499699 / 4.584777 (-4.085078) | 2.991814 / 3.745712 (-0.753898) | 2.906589 / 5.269862 (-2.363272) | 1.842305 / 4.565676 (-2.723372) | 0.057633 / 0.424275 (-0.366642) | 0.006465 / 0.007607 (-0.001142) | 0.492874 / 0.226044 (0.266830) | 4.931613 / 2.268929 (2.662684) | 2.623161 / 55.444624 (-52.821463) | 2.310624 / 6.876477 (-4.565853) | 2.483146 / 2.142072 (0.341074) | 0.586910 / 4.805227 (-4.218317) | 0.124681 / 6.500664 (-6.375983) | 0.061561 / 0.075469 (-0.013908) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.319111 / 1.841788 (-0.522677) | 18.637326 / 8.074308 (10.563018) | 13.803912 / 10.191392 (3.612520) | 0.143989 / 0.680424 (-0.536435) | 0.017025 / 0.534201 (-0.517176) | 0.333156 / 0.579283 (-0.246127) | 0.342163 / 0.434364 (-0.092201) | 0.380357 / 0.540337 (-0.159981) | 0.512261 / 1.386936 (-0.874675) |

</details>
</details>

![](https://cml.dev/watermark.png#f49a16346dc35e5eabeec39778d0f2e4e850dfd7 "CML watermark")