Issue: Set minimal fsspec version requirement to 2023.1.0
State: closed
Author: mariosasko

Fix https://github.com/huggingface/datasets/issues/6141

Colab installs 2023.6.0, so we should be good 🙂


Comments:
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005972 / 0.011353 (-0.005381) | 0.003636 / 0.011008 (-0.007372) | 0.080254 / 0.038508 (0.041746) | 0.059564 / 0.023109 (0.036455) | 0.310615 / 0.275898 (0.034717) | 0.359307 / 0.323480 (0.035827) | 0.003408 / 0.007986 (-0.004578) | 0.002941 / 0.004328 (-0.001388) | 0.063699 / 0.004250 (0.059449) | 0.046072 / 0.037052 (0.009020) | 0.318670 / 0.258489 (0.060181) | 0.369677 / 0.293841 (0.075836) | 0.026995 / 0.128546 (-0.101552) | 0.007954 / 0.075646 (-0.067693) | 0.261667 / 0.419271 (-0.157604) | 0.045167 / 0.043533 (0.001634) | 0.314276 / 0.255139 (0.059137) | 0.348871 / 0.283200 (0.065672) | 0.021748 / 0.141683 (-0.119935) | 1.438598 / 1.452155 (-0.013557) | 1.530119 / 1.492716 (0.037403) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.196894 / 0.018006 (0.178888) | 0.445757 / 0.000490 (0.445267) | 0.002842 / 0.000200 (0.002642) | 0.000069 / 0.000054 (0.000015) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024923 / 0.037411 (-0.012488) | 0.075186 / 0.014526 (0.060661) | 0.087193 / 0.176557 (-0.089364) | 0.147496 / 0.737135 (-0.589639) | 0.087083 / 0.296338 (-0.209255) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.423545 / 0.215209 (0.208336) | 4.187927 / 2.077655 (2.110273) | 2.008656 / 1.504120 (0.504536) | 1.791313 / 1.541195 (0.250119) | 1.849836 / 1.468490 (0.381346) | 0.499458 / 4.584777 (-4.085318) | 2.983206 / 3.745712 (-0.762506) | 2.801005 / 5.269862 (-2.468856) | 1.886207 / 4.565676 (-2.679469) | 0.057343 / 0.424275 (-0.366932) | 0.006666 / 0.007607 (-0.000941) | 0.483948 / 0.226044 (0.257904) | 4.874818 / 2.268929 (2.605890) | 2.439393 / 55.444624 (-53.005231) | 2.049861 / 6.876477 (-4.826616) | 2.217050 / 2.142072 (0.074977) | 0.589760 / 4.805227 (-4.215467) | 0.125298 / 6.500664 (-6.375366) | 0.061123 / 0.075469 (-0.014347) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.234721 / 1.841788 (-0.607067) | 18.193756 / 8.074308 (10.119448) | 13.682835 / 10.191392 (3.491443) | 0.129345 / 0.680424 (-0.551078) | 0.016589 / 0.534201 (-0.517612) | 0.332355 / 0.579283 (-0.246928) | 0.358408 / 0.434364 (-0.075955) | 0.382044 / 0.540337 (-0.158293) | 0.535403 / 1.386936 (-0.851533) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006193 / 0.011353 (-0.005160) | 0.003674 / 0.011008 (-0.007335) | 0.062481 / 0.038508 (0.023973) | 0.062096 / 0.023109 (0.038987) | 0.449592 / 0.275898 (0.173694) | 0.479245 / 0.323480 (0.155765) | 0.004793 / 0.007986 (-0.003193) | 0.002896 / 0.004328 (-0.001433) | 0.062887 / 0.004250 (0.058636) | 0.050049 / 0.037052 (0.012997) | 0.454940 / 0.258489 (0.196451) | 0.486115 / 0.293841 (0.192274) | 0.028585 / 0.128546 (-0.099961) | 0.007954 / 0.075646 (-0.067692) | 0.067744 / 0.419271 (-0.351528) | 0.040473 / 0.043533 (-0.003060) | 0.448408 / 0.255139 (0.193269) | 0.472423 / 0.283200 (0.189223) | 0.020549 / 0.141683 (-0.121133) | 1.563618 / 1.452155 (0.111463) | 1.520149 / 1.492716 (0.027432) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.226604 / 0.018006 (0.208598) | 0.417615 / 0.000490 (0.417126) | 0.003386 / 0.000200 (0.003186) | 0.000074 / 0.000054 (0.000019) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027264 / 0.037411 (-0.010147) | 0.081709 / 0.014526 (0.067184) | 0.091793 / 0.176557 (-0.084763) | 0.145559 / 0.737135 (-0.591576) | 0.091869 / 0.296338 (-0.204469) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.462917 / 0.215209 (0.247708) | 4.629512 / 2.077655 (2.551857) | 2.555715 / 1.504120 (1.051595) | 2.388064 / 1.541195 (0.846870) | 2.458320 / 1.468490 (0.989830) | 0.511615 / 4.584777 (-4.073162) | 3.124566 / 3.745712 (-0.621146) | 2.839190 / 5.269862 (-2.430672) | 1.894551 / 4.565676 (-2.671126) | 0.059565 / 0.424275 (-0.364710) | 0.006481 / 0.007607 (-0.001126) | 0.532023 / 0.226044 (0.305979) | 5.361507 / 2.268929 (3.092579) | 2.982594 / 55.444624 (-52.462031) | 2.644870 / 6.876477 (-4.231606) | 2.831476 / 2.142072 (0.689404) | 0.607381 / 4.805227 (-4.197846) | 0.126067 / 6.500664 (-6.374597) | 0.062130 / 0.075469 (-0.013339) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.350442 / 1.841788 (-0.491345) | 18.829553 / 8.074308 (10.755245) | 14.796701 / 10.191392 (4.605309) | 0.145393 / 0.680424 (-0.535031) | 0.018218 / 0.534201 (-0.515983) | 0.335500 / 0.579283 (-0.243783) | 0.359190 / 0.434364 (-0.075174) | 0.388377 / 0.540337 (-0.151960) | 0.534994 / 1.386936 (-0.851942) |

</details>
</details>

![](https://cml.dev/watermark.png#ff7629eb72f499d841d64aa03f97e0b1707d1cc7 "CML watermark")


_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006741 / 0.011353 (-0.004612) | 0.004097 / 0.011008 (-0.006911) | 0.084513 / 0.038508 (0.046005) | 0.074216 / 0.023109 (0.051107) | 0.352481 / 0.275898 (0.076583) | 0.394806 / 0.323480 (0.071326) | 0.005603 / 0.007986 (-0.002383) | 0.003482 / 0.004328 (-0.000847) | 0.065165 / 0.004250 (0.060914) | 0.054065 / 0.037052 (0.017013) | 0.359399 / 0.258489 (0.100910) | 0.409776 / 0.293841 (0.115935) | 0.030997 / 0.128546 (-0.097550) | 0.008717 / 0.075646 (-0.066929) | 0.288692 / 0.419271 (-0.130579) | 0.052372 / 0.043533 (0.008840) | 0.353867 / 0.255139 (0.098728) | 0.391212 / 0.283200 (0.108012) | 0.024033 / 0.141683 (-0.117650) | 1.496552 / 1.452155 (0.044398) | 1.567267 / 1.492716 (0.074550) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.294074 / 0.018006 (0.276067) | 0.595421 / 0.000490 (0.594931) | 0.003826 / 0.000200 (0.003626) | 0.000085 / 0.000054 (0.000030) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028676 / 0.037411 (-0.008736) | 0.082064 / 0.014526 (0.067538) | 0.542399 / 0.176557 (0.365842) | 0.217188 / 0.737135 (-0.519947) | 0.099364 / 0.296338 (-0.196975) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.384282 / 0.215209 (0.169073) | 3.832204 / 2.077655 (1.754550) | 1.842500 / 1.504120 (0.338380) | 1.668192 / 1.541195 (0.126997) | 1.745207 / 1.468490 (0.276717) | 0.481881 / 4.584777 (-4.102896) | 3.677819 / 3.745712 (-0.067893) | 3.329062 / 5.269862 (-1.940799) | 2.056882 / 4.565676 (-2.508795) | 0.056898 / 0.424275 (-0.367377) | 0.007624 / 0.007607 (0.000016) | 0.459712 / 0.226044 (0.233667) | 4.611100 / 2.268929 (2.342171) | 2.370244 / 55.444624 (-53.074381) | 2.032756 / 6.876477 (-4.843721) | 2.336056 / 2.142072 (0.193984) | 0.583503 / 4.805227 (-4.221725) | 0.135041 / 6.500664 (-6.365623) | 0.062245 / 0.075469 (-0.013224) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.303894 / 1.841788 (-0.537894) | 20.315185 / 8.074308 (12.240876) | 14.388779 / 10.191392 (4.197387) | 0.169060 / 0.680424 (-0.511364) | 0.018609 / 0.534201 (-0.515592) | 0.395140 / 0.579283 (-0.184143) | 0.418231 / 0.434364 (-0.016133) | 0.461496 / 0.540337 (-0.078842) | 0.630298 / 1.386936 (-0.756638) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006999 / 0.011353 (-0.004354) | 0.004197 / 0.011008 (-0.006812) | 0.064524 / 0.038508 (0.026016) | 0.078791 / 0.023109 (0.055682) | 0.397563 / 0.275898 (0.121665) | 0.423056 / 0.323480 (0.099576) | 0.005697 / 0.007986 (-0.002288) | 0.003592 / 0.004328 (-0.000736) | 0.066178 / 0.004250 (0.061928) | 0.058114 / 0.037052 (0.021062) | 0.398619 / 0.258489 (0.140130) | 0.435496 / 0.293841 (0.141655) | 0.032758 / 0.128546 (-0.095788) | 0.008677 / 0.075646 (-0.066970) | 0.071359 / 0.419271 (-0.347913) | 0.048636 / 0.043533 (0.005103) | 0.389762 / 0.255139 (0.134623) | 0.412109 / 0.283200 (0.128910) | 0.023511 / 0.141683 (-0.118172) | 1.514768 / 1.452155 (0.062613) | 1.580163 / 1.492716 (0.087446) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.370491 / 0.018006 (0.352485) | 0.529751 / 0.000490 (0.529261) | 0.016959 / 0.000200 (0.016759) | 0.000112 / 0.000054 (0.000057) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033361 / 0.037411 (-0.004051) | 0.091610 / 0.014526 (0.077084) | 0.106642 / 0.176557 (-0.069915) | 0.160906 / 0.737135 (-0.576229) | 0.106894 / 0.296338 (-0.189444) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.429932 / 0.215209 (0.214723) | 4.276459 / 2.077655 (2.198804) | 2.268518 / 1.504120 (0.764398) | 2.092512 / 1.541195 (0.551317) | 2.182218 / 1.468490 (0.713728) | 0.494464 / 4.584777 (-4.090313) | 3.750731 / 3.745712 (0.005019) | 3.352370 / 5.269862 (-1.917492) | 2.105630 / 4.565676 (-2.460046) | 0.058465 / 0.424275 (-0.365810) | 0.007449 / 0.007607 (-0.000158) | 0.506896 / 0.226044 (0.280851) | 5.070201 / 2.268929 (2.801272) | 2.758128 / 55.444624 (-52.686496) | 2.408378 / 6.876477 (-4.468099) | 2.690633 / 2.142072 (0.548561) | 0.595662 / 4.805227 (-4.209565) | 0.134355 / 6.500664 (-6.366309) | 0.060113 / 0.075469 (-0.015356) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.380413 / 1.841788 (-0.461375) | 20.691210 / 8.074308 (12.616901) | 15.682282 / 10.191392 (5.490890) | 0.165887 / 0.680424 (-0.514536) | 0.020541 / 0.534201 (-0.513660) | 0.397846 / 0.579283 (-0.181437) | 0.425374 / 0.434364 (-0.008990) | 0.476261 / 0.540337 (-0.064076) | 0.648617 / 1.386936 (-0.738319) |

</details>
</details>

![](https://cml.dev/watermark.png#88797b8827334674d7f78c39171c00f0a28ceed6 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008787 / 0.011353 (-0.002566) | 0.007569 / 0.011008 (-0.003439) | 0.103918 / 0.038508 (0.065410) | 0.083347 / 0.023109 (0.060238) | 0.441838 / 0.275898 (0.165940) | 0.420202 / 0.323480 (0.096722) | 0.007295 / 0.007986 (-0.000690) | 0.005366 / 0.004328 (0.001037) | 0.082659 / 0.004250 (0.078409) | 0.059711 / 0.037052 (0.022658) | 0.401821 / 0.258489 (0.143332) | 0.432906 / 0.293841 (0.139065) | 0.048662 / 0.128546 (-0.079885) | 0.014091 / 0.075646 (-0.061555) | 0.352583 / 0.419271 (-0.066689) | 0.064739 / 0.043533 (0.021206) | 0.410890 / 0.255139 (0.155751) | 0.443450 / 0.283200 (0.160251) | 0.035817 / 0.141683 (-0.105866) | 1.754687 / 1.452155 (0.302532) | 1.887338 / 1.492716 (0.394622) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.209440 / 0.018006 (0.191434) | 0.519641 / 0.000490 (0.519152) | 0.005726 / 0.000200 (0.005526) | 0.000107 / 0.000054 (0.000052) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031027 / 0.037411 (-0.006384) | 0.097503 / 0.014526 (0.082977) | 0.106985 / 0.176557 (-0.069572) | 0.178235 / 0.737135 (-0.558900) | 0.108110 / 0.296338 (-0.188228) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.594325 / 0.215209 (0.379116) | 6.159414 / 2.077655 (4.081759) | 2.664892 / 1.504120 (1.160772) | 2.363355 / 1.541195 (0.822160) | 2.410754 / 1.468490 (0.942264) | 0.842557 / 4.584777 (-3.742220) | 5.112059 / 3.745712 (1.366347) | 4.633152 / 5.269862 (-0.636709) | 2.965891 / 4.565676 (-1.599785) | 0.097922 / 0.424275 (-0.326353) | 0.008602 / 0.007607 (0.000995) | 0.773029 / 0.226044 (0.546985) | 7.462314 / 2.268929 (5.193386) | 3.584776 / 55.444624 (-51.859848) | 2.752375 / 6.876477 (-4.124102) | 2.976345 / 2.142072 (0.834272) | 1.049423 / 4.805227 (-3.755804) | 0.212001 / 6.500664 (-6.288663) | 0.074095 / 0.075469 (-0.001374) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.577905 / 1.841788 (-0.263883) | 23.280931 / 8.074308 (15.206623) | 21.017946 / 10.191392 (10.826554) | 0.228746 / 0.680424 (-0.451678) | 0.027877 / 0.534201 (-0.506324) | 0.469173 / 0.579283 (-0.110110) | 0.567614 / 0.434364 (0.133250) | 0.545041 / 0.540337 (0.004704) | 0.754743 / 1.386936 (-0.632194) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008958 / 0.011353 (-0.002395) | 0.005077 / 0.011008 (-0.005931) | 0.083990 / 0.038508 (0.045482) | 0.078586 / 0.023109 (0.055476) | 0.482164 / 0.275898 (0.206266) | 0.525575 / 0.323480 (0.202095) | 0.006031 / 0.007986 (-0.001955) | 0.003922 / 0.004328 (-0.000407) | 0.084547 / 0.004250 (0.080296) | 0.064539 / 0.037052 (0.027487) | 0.501256 / 0.258489 (0.242767) | 0.531985 / 0.293841 (0.238144) | 0.050438 / 0.128546 (-0.078109) | 0.014004 / 0.075646 (-0.061642) | 0.091269 / 0.419271 (-0.328003) | 0.060825 / 0.043533 (0.017292) | 0.492573 / 0.255139 (0.237434) | 0.517060 / 0.283200 (0.233861) | 0.033576 / 0.141683 (-0.108107) | 1.775719 / 1.452155 (0.323564) | 1.866865 / 1.492716 (0.374149) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.225026 / 0.018006 (0.207020) | 0.510715 / 0.000490 (0.510225) | 0.005791 / 0.000200 (0.005591) | 0.000116 / 0.000054 (0.000061) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032795 / 0.037411 (-0.004616) | 0.109206 / 0.014526 (0.094680) | 0.121441 / 0.176557 (-0.055115) | 0.179735 / 0.737135 (-0.557401) | 0.115825 / 0.296338 (-0.180514) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.633259 / 0.215209 (0.418050) | 6.298084 / 2.077655 (4.220430) | 2.892604 / 1.504120 (1.388484) | 2.570858 / 1.541195 (1.029663) | 2.611441 / 1.468490 (1.142951) | 0.897801 / 4.584777 (-3.686976) | 5.185863 / 3.745712 (1.440151) | 4.656897 / 5.269862 (-0.612965) | 3.078575 / 4.565676 (-1.487101) | 0.100563 / 0.424275 (-0.323712) | 0.008368 / 0.007607 (0.000761) | 0.749152 / 0.226044 (0.523108) | 7.687484 / 2.268929 (5.418556) | 3.689238 / 55.444624 (-51.755387) | 2.896779 / 6.876477 (-3.979698) | 3.158688 / 2.142072 (1.016615) | 1.083490 / 4.805227 (-3.721737) | 0.216994 / 6.500664 (-6.283670) | 0.074053 / 0.075469 (-0.001416) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.732812 / 1.841788 (-0.108976) | 23.952127 / 8.074308 (15.877819) | 22.078140 / 10.191392 (11.886748) | 0.229491 / 0.680424 (-0.450933) | 0.032070 / 0.534201 (-0.502131) | 0.503344 / 0.579283 (-0.075939) | 0.588489 / 0.434364 (0.154125) | 0.550199 / 0.540337 (0.009861) | 0.778203 / 1.386936 (-0.608733) |

</details>
</details>

![](https://cml.dev/watermark.png#7e95a508b8d1747b5331bdbbd3e1021e97602c49 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007569 / 0.011353 (-0.003784) | 0.004447 / 0.011008 (-0.006561) | 0.098573 / 0.038508 (0.060064) | 0.081743 / 0.023109 (0.058634) | 0.379912 / 0.275898 (0.104013) | 0.411203 / 0.323480 (0.087723) | 0.004492 / 0.007986 (-0.003494) | 0.005627 / 0.004328 (0.001298) | 0.075974 / 0.004250 (0.071724) | 0.062512 / 0.037052 (0.025459) | 0.386971 / 0.258489 (0.128482) | 0.433299 / 0.293841 (0.139458) | 0.035935 / 0.128546 (-0.092611) | 0.009845 / 0.075646 (-0.065801) | 0.342940 / 0.419271 (-0.076331) | 0.061343 / 0.043533 (0.017810) | 0.381984 / 0.255139 (0.126845) | 0.417921 / 0.283200 (0.134721) | 0.028469 / 0.141683 (-0.113214) | 1.758472 / 1.452155 (0.306317) | 1.847768 / 1.492716 (0.355051) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.234297 / 0.018006 (0.216291) | 0.520020 / 0.000490 (0.519531) | 0.007375 / 0.000200 (0.007175) | 0.000767 / 0.000054 (0.000713) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032738 / 0.037411 (-0.004673) | 0.097656 / 0.014526 (0.083130) | 0.112476 / 0.176557 (-0.064080) | 0.179222 / 0.737135 (-0.557913) | 0.113638 / 0.296338 (-0.182700) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.453677 / 0.215209 (0.238467) | 4.528143 / 2.077655 (2.450489) | 2.243874 / 1.504120 (0.739754) | 2.051546 / 1.541195 (0.510351) | 2.196050 / 1.468490 (0.727560) | 0.567345 / 4.584777 (-4.017432) | 4.133591 / 3.745712 (0.387879) | 3.855286 / 5.269862 (-1.414576) | 2.393496 / 4.565676 (-2.172180) | 0.066567 / 0.424275 (-0.357708) | 0.009038 / 0.007607 (0.001431) | 0.549166 / 0.226044 (0.323122) | 5.472767 / 2.268929 (3.203839) | 2.788012 / 55.444624 (-52.656612) | 2.426132 / 6.876477 (-4.450345) | 2.684856 / 2.142072 (0.542784) | 0.680198 / 4.805227 (-4.125029) | 0.157782 / 6.500664 (-6.342882) | 0.073000 / 0.075469 (-0.002469) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.622435 / 1.841788 (-0.219352) | 22.965715 / 8.074308 (14.891407) | 16.626903 / 10.191392 (6.435511) | 0.197156 / 0.680424 (-0.483268) | 0.025599 / 0.534201 (-0.508602) | 0.495550 / 0.579283 (-0.083733) | 0.466575 / 0.434364 (0.032211) | 0.565862 / 0.540337 (0.025525) | 0.793835 / 1.386936 (-0.593102) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007721 / 0.011353 (-0.003632) | 0.004652 / 0.011008 (-0.006356) | 0.076636 / 0.038508 (0.038127) | 0.082183 / 0.023109 (0.059074) | 0.474665 / 0.275898 (0.198767) | 0.511593 / 0.323480 (0.188113) | 0.006240 / 0.007986 (-0.001746) | 0.003750 / 0.004328 (-0.000578) | 0.076939 / 0.004250 (0.072689) | 0.063333 / 0.037052 (0.026281) | 0.476469 / 0.258489 (0.217980) | 0.512514 / 0.293841 (0.218674) | 0.037802 / 0.128546 (-0.090744) | 0.009975 / 0.075646 (-0.065671) | 0.084190 / 0.419271 (-0.335081) | 0.056705 / 0.043533 (0.013172) | 0.475429 / 0.255139 (0.220290) | 0.496414 / 0.283200 (0.213215) | 0.026039 / 0.141683 (-0.115644) | 1.796059 / 1.452155 (0.343905) | 1.867461 / 1.492716 (0.374745) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.285219 / 0.018006 (0.267213) | 0.506311 / 0.000490 (0.505821) | 0.018545 / 0.000200 (0.018345) | 0.000142 / 0.000054 (0.000088) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.037832 / 0.037411 (0.000420) | 0.110437 / 0.014526 (0.095911) | 0.122953 / 0.176557 (-0.053604) | 0.187049 / 0.737135 (-0.550087) | 0.123539 / 0.296338 (-0.172800) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.508120 / 0.215209 (0.292911) | 5.082836 / 2.077655 (3.005182) | 2.800411 / 1.504120 (1.296291) | 2.579457 / 1.541195 (1.038262) | 2.645945 / 1.468490 (1.177455) | 0.578574 / 4.584777 (-4.006203) | 4.163401 / 3.745712 (0.417689) | 3.858575 / 5.269862 (-1.411286) | 2.389892 / 4.565676 (-2.175785) | 0.068639 / 0.424275 (-0.355636) | 0.008779 / 0.007607 (0.001172) | 0.598925 / 0.226044 (0.372880) | 5.987147 / 2.268929 (3.718219) | 3.361791 / 55.444624 (-52.082833) | 2.910425 / 6.876477 (-3.966051) | 3.156849 / 2.142072 (1.014776) | 0.690945 / 4.805227 (-4.114283) | 0.157441 / 6.500664 (-6.343223) | 0.071596 / 0.075469 (-0.003873) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.672763 / 1.841788 (-0.169025) | 23.599525 / 8.074308 (15.525217) | 17.520087 / 10.191392 (7.328695) | 0.169174 / 0.680424 (-0.511250) | 0.023470 / 0.534201 (-0.510731) | 0.469234 / 0.579283 (-0.110050) | 0.470020 / 0.434364 (0.035656) | 0.579949 / 0.540337 (0.039611) | 0.771353 / 1.386936 (-0.615583) |

</details>
</details>

![](https://cml.dev/watermark.png#029227a116c14720afca71b9b22e78eb2a1c09a6 "CML watermark")