Rename old push_to_hub configs to "default" in dataset_infos
Opened by lhoestq (closed)

Fix

```python
from datasets import load_dataset_builder

b = load_dataset_builder("lambdalabs/pokemon-blip-captions", "default")
print(b.info)
```

which should return

```
DatasetInfo(
  features={'image': Image(decode=True, id=None), 'text': Value(dtype='string', id=None)}, 
  dataset_name='pokemon-blip-captions',
  config_name='default', 
  version=0.0.0,
  splits={'train': SplitInfo(name='train', num_bytes=119417410.0, num_examples=833, shard_lengths=None, dataset_name='pokemon-blip-captions')},
  download_checksums=None,
  download_size=99672355,
  dataset_size=119417410.0,
  size_in_bytes=219089765.0,
  ...
)
```

instead of and empty dataset info.

The dataset has a dataset_infos.json file with a deprecated config name "lambdalabs--pokemon-blip-captions". We switched those config names to "default" in 2.14, so the builder.info should take this into account.

Comments:
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006529 / 0.011353 (-0.004823) | 0.004010 / 0.011008 (-0.006998) | 0.086258 / 0.038508 (0.047750) | 0.073775 / 0.023109 (0.050666) | 0.307573 / 0.275898 (0.031675) | 0.337091 / 0.323480 (0.013611) | 0.004251 / 0.007986 (-0.003735) | 0.003886 / 0.004328 (-0.000443) | 0.068238 / 0.004250 (0.063987) | 0.057000 / 0.037052 (0.019948) | 0.321751 / 0.258489 (0.063262) | 0.359227 / 0.293841 (0.065386) | 0.030841 / 0.128546 (-0.097705) | 0.008569 / 0.075646 (-0.067078) | 0.299523 / 0.419271 (-0.119748) | 0.052563 / 0.043533 (0.009030) | 0.312806 / 0.255139 (0.057667) | 0.342273 / 0.283200 (0.059074) | 0.025725 / 0.141683 (-0.115958) | 1.479263 / 1.452155 (0.027108) | 1.554975 / 1.492716 (0.062259) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.316328 / 0.018006 (0.298322) | 0.598993 / 0.000490 (0.598503) | 0.004548 / 0.000200 (0.004348) | 0.000080 / 0.000054 (0.000026) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027399 / 0.037411 (-0.010013) | 0.081683 / 0.014526 (0.067157) | 0.096968 / 0.176557 (-0.079589) | 0.151559 / 0.737135 (-0.585576) | 0.096558 / 0.296338 (-0.199781) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.383117 / 0.215209 (0.167908) | 3.818634 / 2.077655 (1.740979) | 1.878112 / 1.504120 (0.373992) | 1.729031 / 1.541195 (0.187836) | 1.770259 / 1.468490 (0.301769) | 0.484061 / 4.584777 (-4.100716) | 3.596998 / 3.745712 (-0.148715) | 3.246846 / 5.269862 (-2.023016) | 2.019481 / 4.565676 (-2.546195) | 0.057279 / 0.424275 (-0.366996) | 0.007455 / 0.007607 (-0.000152) | 0.465002 / 0.226044 (0.238958) | 4.644669 / 2.268929 (2.375741) | 2.346415 / 55.444624 (-53.098209) | 2.039686 / 6.876477 (-4.836791) | 2.172822 / 2.142072 (0.030750) | 0.582925 / 4.805227 (-4.222302) | 0.134246 / 6.500664 (-6.366418) | 0.060093 / 0.075469 (-0.015376) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.249033 / 1.841788 (-0.592755) | 19.585949 / 8.074308 (11.511641) | 14.100681 / 10.191392 (3.909289) | 0.147138 / 0.680424 (-0.533286) | 0.018307 / 0.534201 (-0.515894) | 0.397939 / 0.579283 (-0.181344) | 0.413916 / 0.434364 (-0.020448) | 0.465688 / 0.540337 (-0.074650) | 0.642140 / 1.386936 (-0.744797) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006627 / 0.011353 (-0.004726) | 0.004173 / 0.011008 (-0.006835) | 0.063850 / 0.038508 (0.025342) | 0.074733 / 0.023109 (0.051623) | 0.398111 / 0.275898 (0.122213) | 0.426344 / 0.323480 (0.102864) | 0.006261 / 0.007986 (-0.001725) | 0.003507 / 0.004328 (-0.000822) | 0.064511 / 0.004250 (0.060260) | 0.056508 / 0.037052 (0.019456) | 0.401750 / 0.258489 (0.143261) | 0.437081 / 0.293841 (0.143240) | 0.031815 / 0.128546 (-0.096732) | 0.008703 / 0.075646 (-0.066943) | 0.071411 / 0.419271 (-0.347861) | 0.048153 / 0.043533 (0.004620) | 0.399221 / 0.255139 (0.144082) | 0.429312 / 0.283200 (0.146112) | 0.022157 / 0.141683 (-0.119526) | 1.485656 / 1.452155 (0.033502) | 1.550967 / 1.492716 (0.058250) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.330575 / 0.018006 (0.312569) | 0.525553 / 0.000490 (0.525064) | 0.004574 / 0.000200 (0.004374) | 0.000093 / 0.000054 (0.000038) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031871 / 0.037411 (-0.005541) | 0.091819 / 0.014526 (0.077293) | 0.105542 / 0.176557 (-0.071015) | 0.158210 / 0.737135 (-0.578926) | 0.107167 / 0.296338 (-0.189172) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.430226 / 0.215209 (0.215017) | 4.293456 / 2.077655 (2.215801) | 2.289538 / 1.504120 (0.785418) | 2.122255 / 1.541195 (0.581060) | 2.181840 / 1.468490 (0.713350) | 0.498529 / 4.584777 (-4.086248) | 3.686636 / 3.745712 (-0.059077) | 3.287279 / 5.269862 (-1.982582) | 2.068397 / 4.565676 (-2.497280) | 0.058775 / 0.424275 (-0.365500) | 0.007583 / 0.007607 (-0.000024) | 0.507165 / 0.226044 (0.281121) | 5.072330 / 2.268929 (2.803401) | 2.796396 / 55.444624 (-52.648228) | 2.409946 / 6.876477 (-4.466531) | 2.657322 / 2.142072 (0.515250) | 0.597744 / 4.805227 (-4.207483) | 0.133803 / 6.500664 (-6.366861) | 0.060231 / 0.075469 (-0.015238) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.333130 / 1.841788 (-0.508658) | 20.545936 / 8.074308 (12.471627) | 14.875020 / 10.191392 (4.683628) | 0.168873 / 0.680424 (-0.511551) | 0.020316 / 0.534201 (-0.513885) | 0.397203 / 0.579283 (-0.182080) | 0.412412 / 0.434364 (-0.021952) | 0.479952 / 0.540337 (-0.060385) | 0.657155 / 1.386936 (-0.729781) |

</details>
</details>

![](https://cml.dev/watermark.png#13fbee4ca8742460e9baab86a89d9100a294df3e "CML watermark")


_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007885 / 0.011353 (-0.003468) | 0.005221 / 0.011008 (-0.005787) | 0.099457 / 0.038508 (0.060949) | 0.085867 / 0.023109 (0.062758) | 0.359922 / 0.275898 (0.084024) | 0.406479 / 0.323480 (0.082999) | 0.005001 / 0.007986 (-0.002985) | 0.003678 / 0.004328 (-0.000650) | 0.075647 / 0.004250 (0.071396) | 0.064318 / 0.037052 (0.027265) | 0.372180 / 0.258489 (0.113691) | 0.419206 / 0.293841 (0.125365) | 0.040438 / 0.128546 (-0.088108) | 0.010008 / 0.075646 (-0.065638) | 0.340911 / 0.419271 (-0.078360) | 0.063326 / 0.043533 (0.019793) | 0.359015 / 0.255139 (0.103876) | 0.408601 / 0.283200 (0.125402) | 0.029828 / 0.141683 (-0.111855) | 1.767822 / 1.452155 (0.315667) | 1.829079 / 1.492716 (0.336363) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.234455 / 0.018006 (0.216449) | 0.507786 / 0.000490 (0.507297) | 0.004009 / 0.000200 (0.003809) | 0.000101 / 0.000054 (0.000046) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033374 / 0.037411 (-0.004038) | 0.100817 / 0.014526 (0.086291) | 0.113415 / 0.176557 (-0.063141) | 0.180368 / 0.737135 (-0.556768) | 0.115446 / 0.296338 (-0.180893) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.488976 / 0.215209 (0.273767) | 4.911354 / 2.077655 (2.833699) | 2.623525 / 1.504120 (1.119405) | 2.424400 / 1.541195 (0.883206) | 2.497580 / 1.468490 (1.029089) | 0.561106 / 4.584777 (-4.023671) | 4.265649 / 3.745712 (0.519937) | 3.830267 / 5.269862 (-1.439595) | 2.404727 / 4.565676 (-2.160949) | 0.067303 / 0.424275 (-0.356972) | 0.009177 / 0.007607 (0.001570) | 0.588433 / 0.226044 (0.362388) | 5.871573 / 2.268929 (3.602645) | 3.087845 / 55.444624 (-52.356779) | 2.765381 / 6.876477 (-4.111096) | 3.007863 / 2.142072 (0.865791) | 0.687327 / 4.805227 (-4.117901) | 0.157687 / 6.500664 (-6.342977) | 0.071291 / 0.075469 (-0.004178) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.510931 / 1.841788 (-0.330857) | 22.129590 / 8.074308 (14.055282) | 16.780479 / 10.191392 (6.589087) | 0.168297 / 0.680424 (-0.512127) | 0.021294 / 0.534201 (-0.512907) | 0.464535 / 0.579283 (-0.114748) | 0.480041 / 0.434364 (0.045677) | 0.549185 / 0.540337 (0.008848) | 0.739438 / 1.386936 (-0.647498) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007834 / 0.011353 (-0.003518) | 0.004576 / 0.011008 (-0.006432) | 0.073331 / 0.038508 (0.034823) | 0.084688 / 0.023109 (0.061579) | 0.486367 / 0.275898 (0.210469) | 0.523127 / 0.323480 (0.199647) | 0.006278 / 0.007986 (-0.001708) | 0.003792 / 0.004328 (-0.000537) | 0.075416 / 0.004250 (0.071166) | 0.064053 / 0.037052 (0.027001) | 0.491908 / 0.258489 (0.233419) | 0.529177 / 0.293841 (0.235336) | 0.038483 / 0.128546 (-0.090063) | 0.009560 / 0.075646 (-0.066087) | 0.083431 / 0.419271 (-0.335841) | 0.057114 / 0.043533 (0.013581) | 0.486316 / 0.255139 (0.231177) | 0.512384 / 0.283200 (0.229185) | 0.028452 / 0.141683 (-0.113231) | 1.788886 / 1.452155 (0.336731) | 1.893834 / 1.492716 (0.401118) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.343018 / 0.018006 (0.325011) | 0.513673 / 0.000490 (0.513183) | 0.056778 / 0.000200 (0.056578) | 0.001799 / 0.000054 (0.001745) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.038530 / 0.037411 (0.001119) | 0.109286 / 0.014526 (0.094760) | 0.122812 / 0.176557 (-0.053745) | 0.187780 / 0.737135 (-0.549355) | 0.124083 / 0.296338 (-0.172255) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.509839 / 0.215209 (0.294630) | 5.085840 / 2.077655 (3.008186) | 2.746695 / 1.504120 (1.242575) | 2.542283 / 1.541195 (1.001088) | 2.650243 / 1.468490 (1.181753) | 0.592801 / 4.584777 (-3.991976) | 4.316721 / 3.745712 (0.571009) | 3.811672 / 5.269862 (-1.458189) | 2.433982 / 4.565676 (-2.131695) | 0.066861 / 0.424275 (-0.357414) | 0.008633 / 0.007607 (0.001026) | 0.590482 / 0.226044 (0.364437) | 5.923484 / 2.268929 (3.654556) | 3.282293 / 55.444624 (-52.162332) | 2.882716 / 6.876477 (-3.993761) | 3.139581 / 2.142072 (0.997509) | 0.690702 / 4.805227 (-4.114525) | 0.156781 / 6.500664 (-6.343883) | 0.071487 / 0.075469 (-0.003982) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.604557 / 1.841788 (-0.237231) | 24.000026 / 8.074308 (15.925718) | 17.548685 / 10.191392 (7.357293) | 0.174883 / 0.680424 (-0.505541) | 0.023812 / 0.534201 (-0.510389) | 0.473522 / 0.579283 (-0.105761) | 0.494683 / 0.434364 (0.060319) | 0.593352 / 0.540337 (0.053015) | 0.771852 / 1.386936 (-0.615084) |

</details>
</details>

![](https://cml.dev/watermark.png#b61c96a806fa97800bc8a66607fb0c78a5d04146 "CML watermark")


thanks! i wonder if we should also fix (change config name) all the old `dataset_infos.json` on the Hub?

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006388 / 0.011353 (-0.004965) | 0.003876 / 0.011008 (-0.007132) | 0.083960 / 0.038508 (0.045452) | 0.068328 / 0.023109 (0.045219) | 0.337958 / 0.275898 (0.062060) | 0.370783 / 0.323480 (0.047303) | 0.003925 / 0.007986 (-0.004060) | 0.004221 / 0.004328 (-0.000107) | 0.064198 / 0.004250 (0.059947) | 0.052681 / 0.037052 (0.015629) | 0.348890 / 0.258489 (0.090401) | 0.389038 / 0.293841 (0.095197) | 0.031133 / 0.128546 (-0.097413) | 0.008566 / 0.075646 (-0.067080) | 0.288169 / 0.419271 (-0.131102) | 0.053290 / 0.043533 (0.009757) | 0.344654 / 0.255139 (0.089515) | 0.381287 / 0.283200 (0.098087) | 0.022350 / 0.141683 (-0.119333) | 1.459933 / 1.452155 (0.007778) | 1.543097 / 1.492716 (0.050380) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.212592 / 0.018006 (0.194586) | 0.461863 / 0.000490 (0.461373) | 0.003468 / 0.000200 (0.003268) | 0.000084 / 0.000054 (0.000029) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026849 / 0.037411 (-0.010563) | 0.081059 / 0.014526 (0.066533) | 0.093986 / 0.176557 (-0.082571) | 0.150328 / 0.737135 (-0.586807) | 0.094253 / 0.296338 (-0.202085) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.382198 / 0.215209 (0.166989) | 3.813878 / 2.077655 (1.736224) | 1.855686 / 1.504120 (0.351566) | 1.672995 / 1.541195 (0.131800) | 1.697705 / 1.468490 (0.229215) | 0.479920 / 4.584777 (-4.104857) | 3.608305 / 3.745712 (-0.137407) | 3.216712 / 5.269862 (-2.053149) | 1.984781 / 4.565676 (-2.580896) | 0.056801 / 0.424275 (-0.367475) | 0.007499 / 0.007607 (-0.000108) | 0.454155 / 0.226044 (0.228110) | 4.531147 / 2.268929 (2.262218) | 2.296149 / 55.444624 (-53.148475) | 1.968701 / 6.876477 (-4.907775) | 2.144286 / 2.142072 (0.002213) | 0.599254 / 4.805227 (-4.205973) | 0.138150 / 6.500664 (-6.362514) | 0.060118 / 0.075469 (-0.015351) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.282486 / 1.841788 (-0.559301) | 19.127792 / 8.074308 (11.053483) | 14.116521 / 10.191392 (3.925129) | 0.163792 / 0.680424 (-0.516632) | 0.018116 / 0.534201 (-0.516085) | 0.390789 / 0.579283 (-0.188494) | 0.409241 / 0.434364 (-0.025123) | 0.457824 / 0.540337 (-0.082513) | 0.624390 / 1.386936 (-0.762546) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006637 / 0.011353 (-0.004716) | 0.003932 / 0.011008 (-0.007076) | 0.063456 / 0.038508 (0.024948) | 0.070062 / 0.023109 (0.046953) | 0.410570 / 0.275898 (0.134672) | 0.436700 / 0.323480 (0.113220) | 0.005324 / 0.007986 (-0.002662) | 0.003263 / 0.004328 (-0.001065) | 0.063590 / 0.004250 (0.059340) | 0.054823 / 0.037052 (0.017770) | 0.408720 / 0.258489 (0.150231) | 0.441493 / 0.293841 (0.147652) | 0.031655 / 0.128546 (-0.096891) | 0.008421 / 0.075646 (-0.067225) | 0.070657 / 0.419271 (-0.348614) | 0.047370 / 0.043533 (0.003837) | 0.408217 / 0.255139 (0.153078) | 0.422178 / 0.283200 (0.138978) | 0.022282 / 0.141683 (-0.119401) | 1.511417 / 1.452155 (0.059262) | 1.570337 / 1.492716 (0.077620) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.224334 / 0.018006 (0.206327) | 0.447589 / 0.000490 (0.447099) | 0.004227 / 0.000200 (0.004027) | 0.000099 / 0.000054 (0.000045) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030797 / 0.037411 (-0.006615) | 0.091276 / 0.014526 (0.076750) | 0.102665 / 0.176557 (-0.073892) | 0.155423 / 0.737135 (-0.581712) | 0.103779 / 0.296338 (-0.192560) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.434509 / 0.215209 (0.219300) | 4.328910 / 2.077655 (2.251255) | 2.311424 / 1.504120 (0.807304) | 2.138380 / 1.541195 (0.597185) | 2.196293 / 1.468490 (0.727803) | 0.482123 / 4.584777 (-4.102654) | 3.597870 / 3.745712 (-0.147842) | 3.222426 / 5.269862 (-2.047435) | 1.994467 / 4.565676 (-2.571210) | 0.057517 / 0.424275 (-0.366758) | 0.007336 / 0.007607 (-0.000271) | 0.504968 / 0.226044 (0.278923) | 5.047940 / 2.268929 (2.779012) | 2.824014 / 55.444624 (-52.620610) | 2.457762 / 6.876477 (-4.418714) | 2.606970 / 2.142072 (0.464897) | 0.580758 / 4.805227 (-4.224469) | 0.132584 / 6.500664 (-6.368080) | 0.059258 / 0.075469 (-0.016211) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.354386 / 1.841788 (-0.487402) | 19.738147 / 8.074308 (11.663839) | 14.858001 / 10.191392 (4.666609) | 0.166074 / 0.680424 (-0.514350) | 0.020181 / 0.534201 (-0.514020) | 0.398333 / 0.579283 (-0.180950) | 0.406969 / 0.434364 (-0.027395) | 0.474515 / 0.540337 (-0.065822) | 0.649571 / 1.386936 (-0.737365) |

</details>
</details>

![](https://cml.dev/watermark.png#b3ac3b3a9c5f40a29fae71504574cfdeebefe349 "CML watermark")


I would say we should delete all `dataset_infos.json` on the Hub...

@albertvillanova @lhoestq @mariosasko should we really stop supporting it and delete from everywhere?
(bc if not, I've found a bug in updating `dataset_infos.json` with `.push_to_hub` and I'd open a PR to fix it)

We can only delete them for the datasets without namespace and open PRs for the others, so we need to keep supporting them for now