Issue: Bump hfh to 0.11.0
State: closed
Author: lhoestq

to fix errors like

```
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://hub-ci.huggingface.co/api/datasets/__DUMMY_TRANSFORMERS_USER__/...
```

(e.g. from this [failing CI](https://github.com/huggingface/datasets/actions/runs/4428956210/jobs/7769160997))

0.11.0 is the current minimum version in `transformers`

around 5% of users are currently using versions `<0.11.0`

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006334 / 0.011353 (-0.005018) | 0.004447 / 0.011008 (-0.006561) | 0.099287 / 0.038508 (0.060779) | 0.027426 / 0.023109 (0.004317) | 0.322638 / 0.275898 (0.046740) | 0.370501 / 0.323480 (0.047021) | 0.004775 / 0.007986 (-0.003210) | 0.003289 / 0.004328 (-0.001040) | 0.076531 / 0.004250 (0.072280) | 0.037485 / 0.037052 (0.000432) | 0.335634 / 0.258489 (0.077145) | 0.384031 / 0.293841 (0.090190) | 0.031258 / 0.128546 (-0.097288) | 0.011619 / 0.075646 (-0.064027) | 0.326309 / 0.419271 (-0.092963) | 0.042513 / 0.043533 (-0.001020) | 0.340817 / 0.255139 (0.085678) | 0.369846 / 0.283200 (0.086646) | 0.084904 / 0.141683 (-0.056779) | 1.481739 / 1.452155 (0.029584) | 1.566593 / 1.492716 (0.073877) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.186424 / 0.018006 (0.168418) | 0.400879 / 0.000490 (0.400389) | 0.003520 / 0.000200 (0.003320) | 0.000079 / 0.000054 (0.000024) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023287 / 0.037411 (-0.014124) | 0.097767 / 0.014526 (0.083241) | 0.103271 / 0.176557 (-0.073286) | 0.165414 / 0.737135 (-0.571722) | 0.106437 / 0.296338 (-0.189901) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.422711 / 0.215209 (0.207502) | 4.221382 / 2.077655 (2.143727) | 1.906807 / 1.504120 (0.402687) | 1.709595 / 1.541195 (0.168400) | 1.720452 / 1.468490 (0.251962) | 0.699477 / 4.584777 (-3.885300) | 3.415840 / 3.745712 (-0.329873) | 2.835669 / 5.269862 (-2.434192) | 1.501775 / 4.565676 (-3.063901) | 0.082896 / 0.424275 (-0.341379) | 0.012855 / 0.007607 (0.005248) | 0.514373 / 0.226044 (0.288329) | 5.190000 / 2.268929 (2.921071) | 2.302539 / 55.444624 (-53.142086) | 1.963410 / 6.876477 (-4.913067) | 2.020944 / 2.142072 (-0.121128) | 0.805919 / 4.805227 (-3.999308) | 0.150604 / 6.500664 (-6.350060) | 0.065977 / 0.075469 (-0.009492) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.206487 / 1.841788 (-0.635300) | 13.631513 / 8.074308 (5.557205) | 13.800258 / 10.191392 (3.608866) | 0.146914 / 0.680424 (-0.533509) | 0.016454 / 0.534201 (-0.517747) | 0.377752 / 0.579283 (-0.201532) | 0.384312 / 0.434364 (-0.050052) | 0.434912 / 0.540337 (-0.105425) | 0.522507 / 1.386936 (-0.864429) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006328 / 0.011353 (-0.005025) | 0.004406 / 0.011008 (-0.006602) | 0.077951 / 0.038508 (0.039443) | 0.026716 / 0.023109 (0.003607) | 0.337303 / 0.275898 (0.061405) | 0.372036 / 0.323480 (0.048556) | 0.004800 / 0.007986 (-0.003185) | 0.003153 / 0.004328 (-0.001175) | 0.076823 / 0.004250 (0.072573) | 0.035873 / 0.037052 (-0.001179) | 0.340243 / 0.258489 (0.081754) | 0.380183 / 0.293841 (0.086342) | 0.032185 / 0.128546 (-0.096361) | 0.011545 / 0.075646 (-0.064101) | 0.086887 / 0.419271 (-0.332384) | 0.041560 / 0.043533 (-0.001973) | 0.338716 / 0.255139 (0.083577) | 0.363080 / 0.283200 (0.079881) | 0.088375 / 0.141683 (-0.053308) | 1.499004 / 1.452155 (0.046850) | 1.585904 / 1.492716 (0.093188) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.211645 / 0.018006 (0.193639) | 0.403707 / 0.000490 (0.403218) | 0.000415 / 0.000200 (0.000215) | 0.000058 / 0.000054 (0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024972 / 0.037411 (-0.012440) | 0.097996 / 0.014526 (0.083470) | 0.105941 / 0.176557 (-0.070616) | 0.155521 / 0.737135 (-0.581615) | 0.108246 / 0.296338 (-0.188092) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.442316 / 0.215209 (0.227107) | 4.417977 / 2.077655 (2.340322) | 2.078324 / 1.504120 (0.574205) | 1.863678 / 1.541195 (0.322483) | 1.917149 / 1.468490 (0.448659) | 0.697628 / 4.584777 (-3.887149) | 3.412810 / 3.745712 (-0.332902) | 1.866473 / 5.269862 (-3.403389) | 1.155923 / 4.565676 (-3.409754) | 0.082831 / 0.424275 (-0.341444) | 0.012367 / 0.007607 (0.004760) | 0.540018 / 0.226044 (0.313974) | 5.420472 / 2.268929 (3.151544) | 2.508540 / 55.444624 (-52.936084) | 2.166397 / 6.876477 (-4.710080) | 2.153486 / 2.142072 (0.011414) | 0.804860 / 4.805227 (-4.000367) | 0.151178 / 6.500664 (-6.349486) | 0.067870 / 0.075469 (-0.007599) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.310387 / 1.841788 (-0.531400) | 13.908916 / 8.074308 (5.834608) | 14.136895 / 10.191392 (3.945503) | 0.139389 / 0.680424 (-0.541035) | 0.016687 / 0.534201 (-0.517514) | 0.379624 / 0.579283 (-0.199659) | 0.382634 / 0.434364 (-0.051730) | 0.439632 / 0.540337 (-0.100706) | 0.524913 / 1.386936 (-0.862023) |

</details>
</details>

![](https://cml.dev/watermark.png#f8f2143b4ed39b58ed415029e7838d767662da91 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006365 / 0.011353 (-0.004988) | 0.004457 / 0.011008 (-0.006551) | 0.097989 / 0.038508 (0.059481) | 0.027686 / 0.023109 (0.004577) | 0.357412 / 0.275898 (0.081514) | 0.368573 / 0.323480 (0.045093) | 0.004859 / 0.007986 (-0.003127) | 0.003262 / 0.004328 (-0.001066) | 0.076487 / 0.004250 (0.072237) | 0.035526 / 0.037052 (-0.001527) | 0.332862 / 0.258489 (0.074373) | 0.369334 / 0.293841 (0.075493) | 0.030750 / 0.128546 (-0.097796) | 0.011503 / 0.075646 (-0.064143) | 0.323289 / 0.419271 (-0.095982) | 0.042302 / 0.043533 (-0.001231) | 0.334009 / 0.255139 (0.078870) | 0.354150 / 0.283200 (0.070951) | 0.082895 / 0.141683 (-0.058788) | 1.499727 / 1.452155 (0.047572) | 1.574123 / 1.492716 (0.081407) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.192583 / 0.018006 (0.174577) | 0.408136 / 0.000490 (0.407646) | 0.001272 / 0.000200 (0.001072) | 0.000070 / 0.000054 (0.000015) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022883 / 0.037411 (-0.014528) | 0.095710 / 0.014526 (0.081185) | 0.106545 / 0.176557 (-0.070011) | 0.165784 / 0.737135 (-0.571352) | 0.108594 / 0.296338 (-0.187744) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.429483 / 0.215209 (0.214274) | 4.292338 / 2.077655 (2.214683) | 1.917759 / 1.504120 (0.413639) | 1.711489 / 1.541195 (0.170294) | 1.735668 / 1.468490 (0.267178) | 0.707602 / 4.584777 (-3.877175) | 3.369643 / 3.745712 (-0.376070) | 1.874517 / 5.269862 (-3.395344) | 1.248560 / 4.565676 (-3.317117) | 0.083247 / 0.424275 (-0.341028) | 0.012606 / 0.007607 (0.004999) | 0.519342 / 0.226044 (0.293297) | 5.225462 / 2.268929 (2.956533) | 2.433230 / 55.444624 (-53.011394) | 2.006005 / 6.876477 (-4.870471) | 2.093156 / 2.142072 (-0.048916) | 0.809372 / 4.805227 (-3.995855) | 0.151691 / 6.500664 (-6.348973) | 0.066680 / 0.075469 (-0.008789) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.226283 / 1.841788 (-0.615505) | 13.604338 / 8.074308 (5.530030) | 13.953245 / 10.191392 (3.761853) | 0.132904 / 0.680424 (-0.547520) | 0.016420 / 0.534201 (-0.517781) | 0.395316 / 0.579283 (-0.183967) | 0.385003 / 0.434364 (-0.049361) | 0.483303 / 0.540337 (-0.057034) | 0.578459 / 1.386936 (-0.808477) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006218 / 0.011353 (-0.005135) | 0.004451 / 0.011008 (-0.006557) | 0.076892 / 0.038508 (0.038384) | 0.027017 / 0.023109 (0.003908) | 0.356976 / 0.275898 (0.081078) | 0.396083 / 0.323480 (0.072603) | 0.005510 / 0.007986 (-0.002476) | 0.003265 / 0.004328 (-0.001063) | 0.075771 / 0.004250 (0.071521) | 0.037117 / 0.037052 (0.000064) | 0.362181 / 0.258489 (0.103692) | 0.401771 / 0.293841 (0.107931) | 0.032062 / 0.128546 (-0.096484) | 0.011453 / 0.075646 (-0.064194) | 0.085773 / 0.419271 (-0.333498) | 0.041679 / 0.043533 (-0.001854) | 0.355120 / 0.255139 (0.099981) | 0.390170 / 0.283200 (0.106970) | 0.088210 / 0.141683 (-0.053473) | 1.526434 / 1.452155 (0.074279) | 1.586019 / 1.492716 (0.093302) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.196836 / 0.018006 (0.178830) | 0.401161 / 0.000490 (0.400671) | 0.002880 / 0.000200 (0.002680) | 0.000080 / 0.000054 (0.000025) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024445 / 0.037411 (-0.012966) | 0.100187 / 0.014526 (0.085661) | 0.106391 / 0.176557 (-0.070165) | 0.159764 / 0.737135 (-0.577372) | 0.109828 / 0.296338 (-0.186511) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.444228 / 0.215209 (0.229018) | 4.420769 / 2.077655 (2.343114) | 2.069437 / 1.504120 (0.565318) | 1.862587 / 1.541195 (0.321392) | 1.934627 / 1.468490 (0.466137) | 0.699681 / 4.584777 (-3.885095) | 3.352540 / 3.745712 (-0.393172) | 2.613172 / 5.269862 (-2.656689) | 1.445116 / 4.565676 (-3.120561) | 0.083086 / 0.424275 (-0.341189) | 0.012715 / 0.007607 (0.005108) | 0.537450 / 0.226044 (0.311405) | 5.403052 / 2.268929 (3.134123) | 2.506703 / 55.444624 (-52.937921) | 2.170198 / 6.876477 (-4.706279) | 2.201909 / 2.142072 (0.059837) | 0.799555 / 4.805227 (-4.005672) | 0.150825 / 6.500664 (-6.349839) | 0.067234 / 0.075469 (-0.008235) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.293097 / 1.841788 (-0.548691) | 13.817133 / 8.074308 (5.742825) | 14.247231 / 10.191392 (4.055839) | 0.128422 / 0.680424 (-0.552002) | 0.016541 / 0.534201 (-0.517660) | 0.382466 / 0.579283 (-0.196817) | 0.380560 / 0.434364 (-0.053804) | 0.439061 / 0.540337 (-0.101276) | 0.521865 / 1.386936 (-0.865071) |

</details>
</details>

![](https://cml.dev/watermark.png#69e60be438c334919f590512fd664436bd6b3667 "CML watermark")


I also took the liberty of removing `_hf_hub_fixes.py` completely :)

> Do you think this is really necessary and convenient? I would naively say that 5% of the users is not a negligible number...

I think it's ok. Most of them are using old versions of `datasets` anyway.



merging, but lmk if you have other concerns

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006810 / 0.011353 (-0.004543) | 0.004683 / 0.011008 (-0.006325) | 0.100889 / 0.038508 (0.062381) | 0.030135 / 0.023109 (0.007026) | 0.356407 / 0.275898 (0.080509) | 0.389175 / 0.323480 (0.065695) | 0.005358 / 0.007986 (-0.002627) | 0.004760 / 0.004328 (0.000432) | 0.075904 / 0.004250 (0.071654) | 0.040341 / 0.037052 (0.003288) | 0.357363 / 0.258489 (0.098874) | 0.394185 / 0.293841 (0.100344) | 0.031322 / 0.128546 (-0.097224) | 0.011636 / 0.075646 (-0.064010) | 0.327327 / 0.419271 (-0.091944) | 0.042494 / 0.043533 (-0.001039) | 0.338079 / 0.255139 (0.082940) | 0.363388 / 0.283200 (0.080189) | 0.087102 / 0.141683 (-0.054581) | 1.505686 / 1.452155 (0.053531) | 1.562112 / 1.492716 (0.069396) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.203630 / 0.018006 (0.185624) | 0.425986 / 0.000490 (0.425496) | 0.003786 / 0.000200 (0.003586) | 0.000071 / 0.000054 (0.000017) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024138 / 0.037411 (-0.013274) | 0.101752 / 0.014526 (0.087226) | 0.105436 / 0.176557 (-0.071121) | 0.165385 / 0.737135 (-0.571750) | 0.114510 / 0.296338 (-0.181828) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.447561 / 0.215209 (0.232352) | 4.449212 / 2.077655 (2.371557) | 2.169472 / 1.504120 (0.665352) | 1.989025 / 1.541195 (0.447831) | 2.036267 / 1.468490 (0.567776) | 0.698647 / 4.584777 (-3.886130) | 3.483281 / 3.745712 (-0.262431) | 1.949306 / 5.269862 (-3.320555) | 1.290313 / 4.565676 (-3.275363) | 0.083079 / 0.424275 (-0.341196) | 0.012759 / 0.007607 (0.005152) | 0.540944 / 0.226044 (0.314899) | 5.473391 / 2.268929 (3.204463) | 2.632037 / 55.444624 (-52.812587) | 2.327396 / 6.876477 (-4.549081) | 2.428880 / 2.142072 (0.286808) | 0.808918 / 4.805227 (-3.996309) | 0.153283 / 6.500664 (-6.347381) | 0.068325 / 0.075469 (-0.007145) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.212527 / 1.841788 (-0.629260) | 14.306444 / 8.074308 (6.232136) | 14.904980 / 10.191392 (4.713588) | 0.142796 / 0.680424 (-0.537628) | 0.016829 / 0.534201 (-0.517372) | 0.384806 / 0.579283 (-0.194477) | 0.390505 / 0.434364 (-0.043859) | 0.441734 / 0.540337 (-0.098603) | 0.526159 / 1.386936 (-0.860777) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006950 / 0.011353 (-0.004403) | 0.004647 / 0.011008 (-0.006362) | 0.078925 / 0.038508 (0.040417) | 0.028081 / 0.023109 (0.004971) | 0.343420 / 0.275898 (0.067522) | 0.380567 / 0.323480 (0.057087) | 0.005286 / 0.007986 (-0.002700) | 0.004816 / 0.004328 (0.000487) | 0.077332 / 0.004250 (0.073081) | 0.042131 / 0.037052 (0.005078) | 0.345371 / 0.258489 (0.086882) | 0.390232 / 0.293841 (0.096392) | 0.032395 / 0.128546 (-0.096152) | 0.011669 / 0.075646 (-0.063978) | 0.087649 / 0.419271 (-0.331622) | 0.042465 / 0.043533 (-0.001068) | 0.342863 / 0.255139 (0.087724) | 0.368947 / 0.283200 (0.085748) | 0.091725 / 0.141683 (-0.049958) | 1.477435 / 1.452155 (0.025280) | 1.563449 / 1.492716 (0.070733) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.208016 / 0.018006 (0.190010) | 0.428387 / 0.000490 (0.427898) | 0.000443 / 0.000200 (0.000243) | 0.000060 / 0.000054 (0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026963 / 0.037411 (-0.010449) | 0.103854 / 0.014526 (0.089328) | 0.109068 / 0.176557 (-0.067488) | 0.160107 / 0.737135 (-0.577028) | 0.112843 / 0.296338 (-0.183496) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.437161 / 0.215209 (0.221952) | 4.396178 / 2.077655 (2.318523) | 2.067597 / 1.504120 (0.563477) | 1.875247 / 1.541195 (0.334053) | 1.962451 / 1.468490 (0.493961) | 0.701427 / 4.584777 (-3.883350) | 3.459564 / 3.745712 (-0.286148) | 1.959482 / 5.269862 (-3.310380) | 1.191866 / 4.565676 (-3.373810) | 0.083243 / 0.424275 (-0.341032) | 0.012740 / 0.007607 (0.005133) | 0.535236 / 0.226044 (0.309191) | 5.351715 / 2.268929 (3.082786) | 2.490868 / 55.444624 (-52.953756) | 2.195680 / 6.876477 (-4.680797) | 2.233854 / 2.142072 (0.091781) | 0.809041 / 4.805227 (-3.996187) | 0.151498 / 6.500664 (-6.349166) | 0.068297 / 0.075469 (-0.007172) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.303596 / 1.841788 (-0.538192) | 14.712746 / 8.074308 (6.638438) | 14.778412 / 10.191392 (4.587020) | 0.147093 / 0.680424 (-0.533331) | 0.017105 / 0.534201 (-0.517096) | 0.381687 / 0.579283 (-0.197596) | 0.402435 / 0.434364 (-0.031929) | 0.453538 / 0.540337 (-0.086800) | 0.538866 / 1.386936 (-0.848070) |

</details>
</details>

![](https://cml.dev/watermark.png#10f637c3a598c8042865b31f779e315a3da5337e "CML watermark")