Support push_to_hub canonical datasets
Opened by albertvillanova (closed)

Support `push_to_hub` canonical datasets.

This is necessary in the Space to convert script-datasets to Parquet: https://huggingface.co/spaces/albertvillanova/convert-dataset-to-parquet

Note that before this PR, the `repo_id` "dataset_name" was transformed to "user/dataset_name". This behavior was introduced by:
- #6269

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6519). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

nice catch @albertvillanova 

@huggingface/datasets this PR is ready for review.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005306 / 0.011353 (-0.006047) | 0.003454 / 0.011008 (-0.007555) | 0.062157 / 0.038508 (0.023649) | 0.051945 / 0.023109 (0.028835) | 0.241834 / 0.275898 (-0.034064) | 0.265590 / 0.323480 (-0.057890) | 0.003149 / 0.007986 (-0.004837) | 0.002695 / 0.004328 (-0.001633) | 0.049197 / 0.004250 (0.044947) | 0.045576 / 0.037052 (0.008524) | 0.242866 / 0.258489 (-0.015623) | 0.280963 / 0.293841 (-0.012878) | 0.028466 / 0.128546 (-0.100080) | 0.010670 / 0.075646 (-0.064976) | 0.206501 / 0.419271 (-0.212771) | 0.035314 / 0.043533 (-0.008219) | 0.240893 / 0.255139 (-0.014246) | 0.264762 / 0.283200 (-0.018438) | 0.019988 / 0.141683 (-0.121695) | 1.095222 / 1.452155 (-0.356933) | 1.144051 / 1.492716 (-0.348666) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.098034 / 0.018006 (0.080028) | 0.308541 / 0.000490 (0.308051) | 0.000261 / 0.000200 (0.000061) | 0.000059 / 0.000054 (0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018646 / 0.037411 (-0.018766) | 0.062881 / 0.014526 (0.048355) | 0.074062 / 0.176557 (-0.102494) | 0.120860 / 0.737135 (-0.616276) | 0.075388 / 0.296338 (-0.220951) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.282974 / 0.215209 (0.067765) | 2.755589 / 2.077655 (0.677934) | 1.459536 / 1.504120 (-0.044584) | 1.364543 / 1.541195 (-0.176652) | 1.429860 / 1.468490 (-0.038630) | 0.573277 / 4.584777 (-4.011500) | 2.422983 / 3.745712 (-1.322730) | 3.257258 / 5.269862 (-2.012603) | 1.930053 / 4.565676 (-2.635623) | 0.067476 / 0.424275 (-0.356799) | 0.005612 / 0.007607 (-0.001995) | 0.351538 / 0.226044 (0.125494) | 3.380356 / 2.268929 (1.111427) | 1.837887 / 55.444624 (-53.606738) | 1.537994 / 6.876477 (-5.338483) | 1.623630 / 2.142072 (-0.518442) | 0.662652 / 4.805227 (-4.142576) | 0.127074 / 6.500664 (-6.373590) | 0.049311 / 0.075469 (-0.026158) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.151273 / 1.841788 (-0.690515) | 12.766622 / 8.074308 (4.692314) | 10.967610 / 10.191392 (0.776218) | 0.131305 / 0.680424 (-0.549119) | 0.014227 / 0.534201 (-0.519974) | 0.292054 / 0.579283 (-0.287229) | 0.262737 / 0.434364 (-0.171627) | 0.334360 / 0.540337 (-0.205978) | 0.446711 / 1.386936 (-0.940225) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005194 / 0.011353 (-0.006159) | 0.003508 / 0.011008 (-0.007500) | 0.049287 / 0.038508 (0.010779) | 0.052109 / 0.023109 (0.029000) | 0.271501 / 0.275898 (-0.004397) | 0.290959 / 0.323480 (-0.032521) | 0.004347 / 0.007986 (-0.003638) | 0.002659 / 0.004328 (-0.001669) | 0.048769 / 0.004250 (0.044518) | 0.039388 / 0.037052 (0.002336) | 0.272811 / 0.258489 (0.014322) | 0.305632 / 0.293841 (0.011791) | 0.028419 / 0.128546 (-0.100127) | 0.010617 / 0.075646 (-0.065029) | 0.057433 / 0.419271 (-0.361838) | 0.032383 / 0.043533 (-0.011149) | 0.266566 / 0.255139 (0.011427) | 0.290993 / 0.283200 (0.007794) | 0.019939 / 0.141683 (-0.121743) | 1.157623 / 1.452155 (-0.294532) | 1.183298 / 1.492716 (-0.309419) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.099074 / 0.018006 (0.081068) | 0.315282 / 0.000490 (0.314792) | 0.000235 / 0.000200 (0.000035) | 0.000057 / 0.000054 (0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022692 / 0.037411 (-0.014719) | 0.076455 / 0.014526 (0.061929) | 0.089094 / 0.176557 (-0.087462) | 0.126407 / 0.737135 (-0.610728) | 0.089588 / 0.296338 (-0.206750) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.338853 / 0.215209 (0.123644) | 2.809843 / 2.077655 (0.732188) | 1.538262 / 1.504120 (0.034143) | 1.418290 / 1.541195 (-0.122905) | 1.435145 / 1.468490 (-0.033345) | 0.565763 / 4.584777 (-4.019014) | 2.491525 / 3.745712 (-1.254187) | 2.944879 / 5.269862 (-2.324983) | 1.835840 / 4.565676 (-2.729837) | 0.065101 / 0.424275 (-0.359174) | 0.005196 / 0.007607 (-0.002412) | 0.345291 / 0.226044 (0.119247) | 3.399658 / 2.268929 (1.130729) | 1.892321 / 55.444624 (-53.552303) | 1.608293 / 6.876477 (-5.268184) | 1.651188 / 2.142072 (-0.490884) | 0.647806 / 4.805227 (-4.157421) | 0.119318 / 6.500664 (-6.381346) | 0.043058 / 0.075469 (-0.032412) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.983956 / 1.841788 (-0.857831) | 13.516125 / 8.074308 (5.441817) | 11.712571 / 10.191392 (1.521179) | 0.134253 / 0.680424 (-0.546171) | 0.015844 / 0.534201 (-0.518357) | 0.292444 / 0.579283 (-0.286839) | 0.282182 / 0.434364 (-0.152182) | 0.329327 / 0.540337 (-0.211010) | 0.419960 / 1.386936 (-0.966976) |

</details>
</details>

![](https://cml.dev/watermark.png#a887ee78835573f5d80f9e414e8443b4caff3541 "CML watermark")