Support fsspec 2023.10.0
State: closed | Author: albertvillanova (MEMBER)

Fix #6333.

Comments: _The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006013 / 0.011353 (-0.005340) | 0.003647 / 0.011008 (-0.007362) | 0.081781 / 0.038508 (0.043273) | 0.059020 / 0.023109 (0.035911) | 0.321823 / 0.275898 (0.045925) | 0.350159 / 0.323480 (0.026679) | 0.003599 / 0.007986 (-0.004386) | 0.002877 / 0.004328 (-0.001452) | 0.063941 / 0.004250 (0.059690) | 0.049460 / 0.037052 (0.012408) | 0.330185 / 0.258489 (0.071696) | 0.362220 / 0.293841 (0.068379) | 0.027613 / 0.128546 (-0.100934) | 0.007976 / 0.075646 (-0.067670) | 0.263386 / 0.419271 (-0.155885) | 0.045504 / 0.043533 (0.001971) | 0.321172 / 0.255139 (0.066033) | 0.345291 / 0.283200 (0.062091) | 0.023133 / 0.141683 (-0.118550) | 1.435816 / 1.452155 (-0.016339) | 1.557241 / 1.492716 (0.064524) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.222228 / 0.018006 (0.204222) | 0.420008 / 0.000490 (0.419518) | 0.008598 / 0.000200 (0.008398) | 0.000343 / 0.000054 (0.000288) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023725 / 0.037411 (-0.013686) | 0.073023 / 0.014526 (0.058497) | 0.814888 / 0.176557 (0.638332) | 0.294122 / 0.737135 (-0.443013) | 0.088945 / 0.296338 (-0.207393) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.393561 / 0.215209 (0.178352) | 3.946544 / 2.077655 (1.868890) | 1.916476 / 1.504120 (0.412356) | 1.721544 / 1.541195 (0.180349) | 1.768583 / 1.468490 (0.300093) | 0.508067 / 4.584777 (-4.076710) | 3.047832 / 3.745712 (-0.697880) | 2.952842 / 5.269862 (-2.317020) | 1.869337 / 4.565676 (-2.696339) | 0.057812 / 0.424275 (-0.366463) | 0.006694 / 0.007607 (-0.000913) | 0.463007 / 0.226044 (0.236963) | 4.635087 / 2.268929 (2.366158) | 2.419833 / 55.444624 (-53.024792) | 2.018519 / 6.876477 (-4.857958) | 2.043430 / 2.142072 (-0.098643) | 0.590895 / 4.805227 (-4.214333) | 0.126113 / 6.500664 (-6.374552) | 0.061045 / 0.075469 (-0.014424) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.226850 / 1.841788 (-0.614937) | 17.336630 / 8.074308 (9.262322) | 13.651049 / 10.191392 (3.459656) | 0.143308 / 0.680424 (-0.537116) | 0.016938 / 0.534201 (-0.517263) | 0.332829 / 0.579283 (-0.246454) | 0.368684 / 0.434364 (-0.065680) | 0.385848 / 0.540337 (-0.154489) | 0.546391 / 1.386936 (-0.840545) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006149 / 0.011353 (-0.005204) | 0.003818 / 0.011008 (-0.007191) | 0.064012 / 0.038508 (0.025504) | 0.059846 / 0.023109 (0.036737) | 0.455928 / 0.275898 (0.180030) | 0.480736 / 0.323480 (0.157256) | 0.004874 / 0.007986 (-0.003111) | 0.002877 / 0.004328 (-0.001451) | 0.064195 / 0.004250 (0.059944) | 0.048146 / 0.037052 (0.011094) | 0.452638 / 0.258489 (0.194149) | 0.484339 / 0.293841 (0.190499) | 0.028832 / 0.128546 (-0.099715) | 0.008162 / 0.075646 (-0.067485) | 0.069855 / 0.419271 (-0.349417) | 0.041429 / 0.043533 (-0.002104) | 0.453282 / 0.255139 (0.198143) | 0.473812 / 0.283200 (0.190613) | 0.021186 / 0.141683 (-0.120497) | 1.465207 / 1.452155 (0.013052) | 1.508216 / 1.492716 (0.015500) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.242491 / 0.018006 (0.224485) | 0.421219 / 0.000490 (0.420730) | 0.011201 / 0.000200 (0.011001) | 0.000083 / 0.000054 (0.000028) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027015 / 0.037411 (-0.010396) | 0.080465 / 0.014526 (0.065939) | 0.092622 / 0.176557 (-0.083934) | 0.146111 / 0.737135 (-0.591024) | 0.091546 / 0.296338 (-0.204793) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.458351 / 0.215209 (0.243142) | 4.591454 / 2.077655 (2.513799) | 2.508156 / 1.504120 (1.004037) | 2.328771 / 1.541195 (0.787576) | 2.423251 / 1.468490 (0.954761) | 0.508504 / 4.584777 (-4.076273) | 3.133789 / 3.745712 (-0.611923) | 2.862777 / 5.269862 (-2.407084) | 1.886327 / 4.565676 (-2.679350) | 0.058017 / 0.424275 (-0.366258) | 0.006496 / 0.007607 (-0.001111) | 0.529629 / 0.226044 (0.303585) | 5.310338 / 2.268929 (3.041409) | 2.973075 / 55.444624 (-52.471549) | 2.601313 / 6.876477 (-4.275163) | 2.777348 / 2.142072 (0.635275) | 0.593711 / 4.805227 (-4.211516) | 0.125453 / 6.500664 (-6.375211) | 0.061034 / 0.075469 (-0.014435) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.374391 / 1.841788 (-0.467397) | 18.768026 / 8.074308 (10.693718) | 15.053637 / 10.191392 (4.862245) | 0.158253 / 0.680424 (-0.522171) | 0.018126 / 0.534201 (-0.516075) | 0.337427 / 0.579283 (-0.241856) | 0.391678 / 0.434364 (-0.042686) | 0.398524 / 0.540337 (-0.141813) | 0.558629 / 1.386936 (-0.828307) |

</details>
</details>

![](https://cml.dev/watermark.png#e0b79660f180c88517884f831eca620bc46a0957 "CML watermark")


I think https://github.com/huggingface/datasets/pull/6334 fixes it already no ?

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006432 / 0.011353 (-0.004921) | 0.003861 / 0.011008 (-0.007147) | 0.084132 / 0.038508 (0.045624) | 0.069391 / 0.023109 (0.046282) | 0.341081 / 0.275898 (0.065183) | 0.375975 / 0.323480 (0.052495) | 0.003962 / 0.007986 (-0.004024) | 0.003235 / 0.004328 (-0.001094) | 0.064927 / 0.004250 (0.060677) | 0.054190 / 0.037052 (0.017137) | 0.350719 / 0.258489 (0.092230) | 0.393216 / 0.293841 (0.099375) | 0.031002 / 0.128546 (-0.097544) | 0.008416 / 0.075646 (-0.067230) | 0.289268 / 0.419271 (-0.130003) | 0.052167 / 0.043533 (0.008634) | 0.347559 / 0.255139 (0.092420) | 0.370908 / 0.283200 (0.087709) | 0.022540 / 0.141683 (-0.119142) | 1.486297 / 1.452155 (0.034143) | 1.576968 / 1.492716 (0.084252) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.237048 / 0.018006 (0.219042) | 0.452065 / 0.000490 (0.451575) | 0.013963 / 0.000200 (0.013763) | 0.000242 / 0.000054 (0.000188) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028084 / 0.037411 (-0.009327) | 0.081271 / 0.014526 (0.066745) | 0.096490 / 0.176557 (-0.080067) | 0.152106 / 0.737135 (-0.585030) | 0.096174 / 0.296338 (-0.200164) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.386585 / 0.215209 (0.171375) | 3.854996 / 2.077655 (1.777342) | 1.832898 / 1.504120 (0.328778) | 1.662832 / 1.541195 (0.121638) | 1.730753 / 1.468490 (0.262263) | 0.485286 / 4.584777 (-4.099491) | 3.571410 / 3.745712 (-0.174302) | 3.373035 / 5.269862 (-1.896826) | 1.995570 / 4.565676 (-2.570107) | 0.056711 / 0.424275 (-0.367564) | 0.007447 / 0.007607 (-0.000160) | 0.462985 / 0.226044 (0.236941) | 4.617186 / 2.268929 (2.348257) | 2.313915 / 55.444624 (-53.130709) | 1.961697 / 6.876477 (-4.914780) | 1.990410 / 2.142072 (-0.151662) | 0.580536 / 4.805227 (-4.224692) | 0.146275 / 6.500664 (-6.354389) | 0.059458 / 0.075469 (-0.016011) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.274841 / 1.841788 (-0.566947) | 18.641853 / 8.074308 (10.567545) | 13.977525 / 10.191392 (3.786133) | 0.151469 / 0.680424 (-0.528955) | 0.018111 / 0.534201 (-0.516090) | 0.393243 / 0.579283 (-0.186040) | 0.412310 / 0.434364 (-0.022054) | 0.461646 / 0.540337 (-0.078692) | 0.633016 / 1.386936 (-0.753920) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006496 / 0.011353 (-0.004857) | 0.003973 / 0.011008 (-0.007035) | 0.064527 / 0.038508 (0.026019) | 0.069390 / 0.023109 (0.046281) | 0.401162 / 0.275898 (0.125264) | 0.431031 / 0.323480 (0.107551) | 0.005244 / 0.007986 (-0.002741) | 0.003283 / 0.004328 (-0.001046) | 0.064931 / 0.004250 (0.060680) | 0.054402 / 0.037052 (0.017350) | 0.397917 / 0.258489 (0.139428) | 0.436728 / 0.293841 (0.142887) | 0.031932 / 0.128546 (-0.096614) | 0.008557 / 0.075646 (-0.067089) | 0.073336 / 0.419271 (-0.345935) | 0.047559 / 0.043533 (0.004026) | 0.395825 / 0.255139 (0.140686) | 0.423002 / 0.283200 (0.139802) | 0.021708 / 0.141683 (-0.119975) | 1.501140 / 1.452155 (0.048985) | 1.558376 / 1.492716 (0.065660) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.289522 / 0.018006 (0.271516) | 0.449078 / 0.000490 (0.448589) | 0.034174 / 0.000200 (0.033974) | 0.000396 / 0.000054 (0.000342) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032533 / 0.037411 (-0.004878) | 0.093398 / 0.014526 (0.078872) | 0.106930 / 0.176557 (-0.069626) | 0.158743 / 0.737135 (-0.578393) | 0.106904 / 0.296338 (-0.189435) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.427479 / 0.215209 (0.212270) | 4.271758 / 2.077655 (2.194103) | 2.298770 / 1.504120 (0.794650) | 2.134906 / 1.541195 (0.593712) | 2.220487 / 1.468490 (0.751996) | 0.490506 / 4.584777 (-4.094270) | 3.593876 / 3.745712 (-0.151836) | 3.225656 / 5.269862 (-2.044205) | 2.004434 / 4.565676 (-2.561243) | 0.058015 / 0.424275 (-0.366260) | 0.007221 / 0.007607 (-0.000387) | 0.504928 / 0.226044 (0.278884) | 5.049547 / 2.268929 (2.780618) | 2.743843 / 55.444624 (-52.700781) | 2.398399 / 6.876477 (-4.478078) | 2.562939 / 2.142072 (0.420867) | 0.597229 / 4.805227 (-4.207998) | 0.134664 / 6.500664 (-6.366001) | 0.059612 / 0.075469 (-0.015857) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.369692 / 1.841788 (-0.472095) | 19.065326 / 8.074308 (10.991018) | 14.404508 / 10.191392 (4.213116) | 0.175809 / 0.680424 (-0.504615) | 0.020137 / 0.534201 (-0.514064) | 0.394043 / 0.579283 (-0.185240) | 0.424772 / 0.434364 (-0.009592) | 0.475587 / 0.540337 (-0.064751) | 0.644275 / 1.386936 (-0.742661) |

</details>
</details>

![](https://cml.dev/watermark.png#224977971accd63d97ba0a90cc108c4754055ebb "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007259 / 0.011353 (-0.004094) | 0.004396 / 0.011008 (-0.006612) | 0.096456 / 0.038508 (0.057948) | 0.078752 / 0.023109 (0.055643) | 0.359215 / 0.275898 (0.083317) | 0.396927 / 0.323480 (0.073448) | 0.005611 / 0.007986 (-0.002375) | 0.003687 / 0.004328 (-0.000641) | 0.072794 / 0.004250 (0.068544) | 0.059794 / 0.037052 (0.022741) | 0.372352 / 0.258489 (0.113863) | 0.414038 / 0.293841 (0.120197) | 0.034490 / 0.128546 (-0.094056) | 0.009790 / 0.075646 (-0.065857) | 0.326338 / 0.419271 (-0.092934) | 0.058582 / 0.043533 (0.015049) | 0.354221 / 0.255139 (0.099082) | 0.386669 / 0.283200 (0.103469) | 0.025356 / 0.141683 (-0.116327) | 1.664104 / 1.452155 (0.211950) | 1.766825 / 1.492716 (0.274108) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.251107 / 0.018006 (0.233101) | 0.478833 / 0.000490 (0.478344) | 0.010776 / 0.000200 (0.010577) | 0.000292 / 0.000054 (0.000238) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032869 / 0.037411 (-0.004543) | 0.098449 / 0.014526 (0.083923) | 0.109954 / 0.176557 (-0.066602) | 0.176786 / 0.737135 (-0.560350) | 0.113477 / 0.296338 (-0.182862) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.431169 / 0.215209 (0.215960) | 4.303239 / 2.077655 (2.225585) | 2.088885 / 1.504120 (0.584765) | 1.895900 / 1.541195 (0.354706) | 1.997442 / 1.468490 (0.528952) | 0.541840 / 4.584777 (-4.042937) | 3.991982 / 3.745712 (0.246270) | 3.842421 / 5.269862 (-1.427440) | 2.281150 / 4.565676 (-2.284526) | 0.063851 / 0.424275 (-0.360425) | 0.008470 / 0.007607 (0.000863) | 0.515886 / 0.226044 (0.289841) | 5.202908 / 2.268929 (2.933980) | 2.662789 / 55.444624 (-52.781835) | 2.266731 / 6.876477 (-4.609746) | 2.343760 / 2.142072 (0.201688) | 0.641050 / 4.805227 (-4.164177) | 0.148236 / 6.500664 (-6.352428) | 0.067422 / 0.075469 (-0.008047) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.475729 / 1.841788 (-0.366059) | 22.401583 / 8.074308 (14.327274) | 15.886237 / 10.191392 (5.694845) | 0.171828 / 0.680424 (-0.508595) | 0.022161 / 0.534201 (-0.512040) | 0.465873 / 0.579283 (-0.113411) | 0.476386 / 0.434364 (0.042022) | 0.538317 / 0.540337 (-0.002020) | 0.754375 / 1.386936 (-0.632561) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007429 / 0.011353 (-0.003924) | 0.004592 / 0.011008 (-0.006416) | 0.072315 / 0.038508 (0.033807) | 0.080806 / 0.023109 (0.057697) | 0.444607 / 0.275898 (0.168709) | 0.476970 / 0.323480 (0.153490) | 0.006030 / 0.007986 (-0.001956) | 0.003755 / 0.004328 (-0.000573) | 0.074602 / 0.004250 (0.070352) | 0.061846 / 0.037052 (0.024794) | 0.450928 / 0.258489 (0.192439) | 0.493932 / 0.293841 (0.200091) | 0.037398 / 0.128546 (-0.091148) | 0.009807 / 0.075646 (-0.065840) | 0.080531 / 0.419271 (-0.338741) | 0.054052 / 0.043533 (0.010519) | 0.453034 / 0.255139 (0.197895) | 0.464959 / 0.283200 (0.181760) | 0.024718 / 0.141683 (-0.116965) | 1.687552 / 1.452155 (0.235397) | 1.765746 / 1.492716 (0.273029) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.266998 / 0.018006 (0.248992) | 0.479832 / 0.000490 (0.479342) | 0.005429 / 0.000200 (0.005229) | 0.000117 / 0.000054 (0.000062) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.038885 / 0.037411 (0.001474) | 0.105931 / 0.014526 (0.091405) | 0.120880 / 0.176557 (-0.055677) | 0.184006 / 0.737135 (-0.553130) | 0.120750 / 0.296338 (-0.175589) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.478626 / 0.215209 (0.263417) | 4.797355 / 2.077655 (2.719700) | 2.582758 / 1.504120 (1.078638) | 2.396488 / 1.541195 (0.855293) | 2.515597 / 1.468490 (1.047107) | 0.544541 / 4.584777 (-4.040236) | 4.150702 / 3.745712 (0.404990) | 3.676837 / 5.269862 (-1.593024) | 2.287275 / 4.565676 (-2.278402) | 0.064602 / 0.424275 (-0.359673) | 0.008253 / 0.007607 (0.000646) | 0.576201 / 0.226044 (0.350157) | 5.859839 / 2.268929 (3.590910) | 3.248603 / 55.444624 (-52.196021) | 2.841959 / 6.876477 (-4.034518) | 2.991120 / 2.142072 (0.849047) | 0.667755 / 4.805227 (-4.137472) | 0.151219 / 6.500664 (-6.349445) | 0.068990 / 0.075469 (-0.006479) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.572359 / 1.841788 (-0.269429) | 21.890279 / 8.074308 (13.815971) | 15.927473 / 10.191392 (5.736081) | 0.170388 / 0.680424 (-0.510036) | 0.023282 / 0.534201 (-0.510919) | 0.459371 / 0.579283 (-0.119912) | 0.468838 / 0.434364 (0.034475) | 0.546438 / 0.540337 (0.006101) | 0.746912 / 1.386936 (-0.640024) |

</details>
</details>

![](https://cml.dev/watermark.png#8197ce872d2e24bd1ffbb07213faea25078f1386 "CML watermark")


Yes, @lhoestq, you are right. I think we cross-send fixing PRs in a 15 minute interval... :sweat_smile: 

I would say the code in this PR is simpler and easier to understand, but feel free to ignore it.

I think the correct way it to check if "file" in in the tuple if it's a tuple (in case someone adds another protocol name for the local filesystem)