More robust temporary directory deletion
Opened by mariosasko (closed)

While fixing the Windows errors in #6362, I noticed that `PermissionError` can still easily be thrown on the session exit by the temporary cache directory's finalizer (we would also have to keep track of intermediate datasets, copies, etc.). ~~Due to the low usage of `datasets` on Windows, this PR takes a simpler approach to the issue than https://github.com/huggingface/datasets/pull/2403 - it tries to delete the temporary cache directory, and if this fails, logs a warning message about using a `delete-temp-cache` CLI command to delete it manually. The problematic references are freed after the session exits, so the CLI command should then succeed.~~ This PR implements `Dataset.__setstate__` to register datasets with temporary cache files for deletion. 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6426). All of your documentation changes will be reflected on that endpoint.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004750 / 0.011353 (-0.006603) | 0.002928 / 0.011008 (-0.008080) | 0.061962 / 0.038508 (0.023454) | 0.029878 / 0.023109 (0.006768) | 0.233380 / 0.275898 (-0.042518) | 0.262221 / 0.323480 (-0.061259) | 0.002982 / 0.007986 (-0.005004) | 0.003698 / 0.004328 (-0.000630) | 0.048565 / 0.004250 (0.044314) | 0.046107 / 0.037052 (0.009055) | 0.240090 / 0.258489 (-0.018399) | 0.267294 / 0.293841 (-0.026547) | 0.023335 / 0.128546 (-0.105211) | 0.007221 / 0.075646 (-0.068425) | 0.200903 / 0.419271 (-0.218369) | 0.059237 / 0.043533 (0.015705) | 0.234929 / 0.255139 (-0.020210) | 0.256326 / 0.283200 (-0.026874) | 0.018549 / 0.141683 (-0.123134) | 1.103519 / 1.452155 (-0.348635) | 1.156573 / 1.492716 (-0.336143) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091205 / 0.018006 (0.073199) | 0.303533 / 0.000490 (0.303043) | 0.000204 / 0.000200 (0.000004) | 0.000042 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018572 / 0.037411 (-0.018839) | 0.062323 / 0.014526 (0.047797) | 0.074528 / 0.176557 (-0.102029) | 0.120295 / 0.737135 (-0.616841) | 0.076786 / 0.296338 (-0.219552) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.278814 / 0.215209 (0.063605) | 2.745483 / 2.077655 (0.667829) | 1.486073 / 1.504120 (-0.018047) | 1.385334 / 1.541195 (-0.155861) | 1.386351 / 1.468490 (-0.082139) | 0.395545 / 4.584777 (-4.189232) | 2.409468 / 3.745712 (-1.336244) | 2.670702 / 5.269862 (-2.599159) | 1.629245 / 4.565676 (-2.936432) | 0.045990 / 0.424275 (-0.378286) | 0.004782 / 0.007607 (-0.002825) | 0.332912 / 0.226044 (0.106867) | 3.249277 / 2.268929 (0.980349) | 1.888690 / 55.444624 (-53.555934) | 1.533462 / 6.876477 (-5.343015) | 1.576045 / 2.142072 (-0.566027) | 0.473090 / 4.805227 (-4.332138) | 0.099448 / 6.500664 (-6.401216) | 0.042613 / 0.075469 (-0.032857) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.944229 / 1.841788 (-0.897559) | 12.103621 / 8.074308 (4.029313) | 10.643471 / 10.191392 (0.452079) | 0.143004 / 0.680424 (-0.537420) | 0.013872 / 0.534201 (-0.520329) | 0.272026 / 0.579283 (-0.307257) | 0.298701 / 0.434364 (-0.135663) | 0.310299 / 0.540337 (-0.230038) | 0.420934 / 1.386936 (-0.966002) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004904 / 0.011353 (-0.006449) | 0.003064 / 0.011008 (-0.007945) | 0.047982 / 0.038508 (0.009474) | 0.056354 / 0.023109 (0.033245) | 0.292893 / 0.275898 (0.016995) | 0.348744 / 0.323480 (0.025264) | 0.003988 / 0.007986 (-0.003997) | 0.002431 / 0.004328 (-0.001898) | 0.049108 / 0.004250 (0.044857) | 0.039055 / 0.037052 (0.002002) | 0.278129 / 0.258489 (0.019640) | 0.318547 / 0.293841 (0.024706) | 0.025040 / 0.128546 (-0.103507) | 0.007166 / 0.075646 (-0.068480) | 0.053967 / 0.419271 (-0.365305) | 0.033128 / 0.043533 (-0.010405) | 0.272849 / 0.255139 (0.017710) | 0.312143 / 0.283200 (0.028943) | 0.017942 / 0.141683 (-0.123741) | 1.192297 / 1.452155 (-0.259857) | 1.328102 / 1.492716 (-0.164615) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090903 / 0.018006 (0.072896) | 0.301260 / 0.000490 (0.300770) | 0.000215 / 0.000200 (0.000015) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021112 / 0.037411 (-0.016300) | 0.070181 / 0.014526 (0.055656) | 0.082431 / 0.176557 (-0.094126) | 0.121973 / 0.737135 (-0.615163) | 0.083617 / 0.296338 (-0.212721) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.289587 / 0.215209 (0.074378) | 2.877895 / 2.077655 (0.800240) | 1.721417 / 1.504120 (0.217297) | 1.536023 / 1.541195 (-0.005171) | 1.550917 / 1.468490 (0.082427) | 0.402978 / 4.584777 (-4.181799) | 2.431767 / 3.745712 (-1.313946) | 2.544419 / 5.269862 (-2.725442) | 1.554562 / 4.565676 (-3.011115) | 0.046260 / 0.424275 (-0.378015) | 0.004923 / 0.007607 (-0.002684) | 0.341584 / 0.226044 (0.115540) | 3.362133 / 2.268929 (1.093205) | 1.928741 / 55.444624 (-53.515884) | 1.654798 / 6.876477 (-5.221679) | 1.715111 / 2.142072 (-0.426962) | 0.471029 / 4.805227 (-4.334198) | 0.098912 / 6.500664 (-6.401752) | 0.041018 / 0.075469 (-0.034451) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.992880 / 1.841788 (-0.848907) | 12.083890 / 8.074308 (4.009582) | 11.023833 / 10.191392 (0.832441) | 0.139217 / 0.680424 (-0.541207) | 0.015183 / 0.534201 (-0.519018) | 0.271637 / 0.579283 (-0.307646) | 0.278910 / 0.434364 (-0.155454) | 0.306891 / 0.540337 (-0.233447) | 0.424412 / 1.386936 (-0.962524) |

</details>
</details>

![](https://cml.dev/watermark.png#2d51f37eb9996d4c52250ee6e987ccce0d74f2f4 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004545 / 0.011353 (-0.006808) | 0.002955 / 0.011008 (-0.008054) | 0.062119 / 0.038508 (0.023611) | 0.029357 / 0.023109 (0.006248) | 0.240068 / 0.275898 (-0.035830) | 0.273376 / 0.323480 (-0.050104) | 0.003884 / 0.007986 (-0.004102) | 0.002390 / 0.004328 (-0.001938) | 0.048621 / 0.004250 (0.044371) | 0.043867 / 0.037052 (0.006815) | 0.247240 / 0.258489 (-0.011249) | 0.279187 / 0.293841 (-0.014654) | 0.023377 / 0.128546 (-0.105169) | 0.007261 / 0.075646 (-0.068385) | 0.201913 / 0.419271 (-0.217359) | 0.057063 / 0.043533 (0.013530) | 0.245698 / 0.255139 (-0.009441) | 0.265644 / 0.283200 (-0.017556) | 0.018077 / 0.141683 (-0.123606) | 1.133225 / 1.452155 (-0.318930) | 1.186380 / 1.492716 (-0.306336) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.089639 / 0.018006 (0.071632) | 0.298918 / 0.000490 (0.298428) | 0.000198 / 0.000200 (-0.000002) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019037 / 0.037411 (-0.018374) | 0.062580 / 0.014526 (0.048055) | 0.072974 / 0.176557 (-0.103582) | 0.119909 / 0.737135 (-0.617226) | 0.075021 / 0.296338 (-0.221317) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.276561 / 0.215209 (0.061352) | 2.697281 / 2.077655 (0.619626) | 1.419772 / 1.504120 (-0.084348) | 1.302079 / 1.541195 (-0.239115) | 1.329143 / 1.468490 (-0.139347) | 0.395528 / 4.584777 (-4.189249) | 2.365788 / 3.745712 (-1.379925) | 2.583802 / 5.269862 (-2.686059) | 1.561983 / 4.565676 (-3.003694) | 0.045269 / 0.424275 (-0.379006) | 0.004826 / 0.007607 (-0.002781) | 0.331041 / 0.226044 (0.104996) | 3.292523 / 2.268929 (1.023595) | 1.797865 / 55.444624 (-53.646759) | 1.509229 / 6.876477 (-5.367248) | 1.498884 / 2.142072 (-0.643188) | 0.458518 / 4.805227 (-4.346709) | 0.098076 / 6.500664 (-6.402588) | 0.042290 / 0.075469 (-0.033179) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.922331 / 1.841788 (-0.919457) | 11.605041 / 8.074308 (3.530732) | 10.471664 / 10.191392 (0.280272) | 0.130325 / 0.680424 (-0.550098) | 0.014084 / 0.534201 (-0.520117) | 0.278877 / 0.579283 (-0.300406) | 0.263104 / 0.434364 (-0.171259) | 0.306723 / 0.540337 (-0.233615) | 0.416238 / 1.386936 (-0.970698) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005094 / 0.011353 (-0.006259) | 0.002794 / 0.011008 (-0.008214) | 0.048189 / 0.038508 (0.009680) | 0.050409 / 0.023109 (0.027300) | 0.272618 / 0.275898 (-0.003280) | 0.293589 / 0.323480 (-0.029891) | 0.003995 / 0.007986 (-0.003991) | 0.002373 / 0.004328 (-0.001956) | 0.048269 / 0.004250 (0.044018) | 0.038751 / 0.037052 (0.001698) | 0.273495 / 0.258489 (0.015006) | 0.309244 / 0.293841 (0.015403) | 0.024681 / 0.128546 (-0.103866) | 0.007390 / 0.075646 (-0.068256) | 0.053844 / 0.419271 (-0.365427) | 0.032395 / 0.043533 (-0.011137) | 0.271963 / 0.255139 (0.016824) | 0.289557 / 0.283200 (0.006357) | 0.018659 / 0.141683 (-0.123024) | 1.154478 / 1.452155 (-0.297676) | 1.199772 / 1.492716 (-0.292944) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.089771 / 0.018006 (0.071764) | 0.299468 / 0.000490 (0.298978) | 0.000219 / 0.000200 (0.000020) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021854 / 0.037411 (-0.015558) | 0.070280 / 0.014526 (0.055754) | 0.080956 / 0.176557 (-0.095600) | 0.119430 / 0.737135 (-0.617705) | 0.082778 / 0.296338 (-0.213561) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.304273 / 0.215209 (0.089064) | 2.968264 / 2.077655 (0.890609) | 1.592363 / 1.504120 (0.088243) | 1.460795 / 1.541195 (-0.080400) | 1.501545 / 1.468490 (0.033055) | 0.411001 / 4.584777 (-4.173776) | 2.464273 / 3.745712 (-1.281439) | 2.524585 / 5.269862 (-2.745277) | 1.537443 / 4.565676 (-3.028234) | 0.046163 / 0.424275 (-0.378112) | 0.004783 / 0.007607 (-0.002824) | 0.354251 / 0.226044 (0.128206) | 3.512087 / 2.268929 (1.243158) | 1.968156 / 55.444624 (-53.476468) | 1.664966 / 6.876477 (-5.211510) | 1.685013 / 2.142072 (-0.457060) | 0.485793 / 4.805227 (-4.319435) | 0.099789 / 6.500664 (-6.400875) | 0.040705 / 0.075469 (-0.034764) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.966570 / 1.841788 (-0.875218) | 12.023188 / 8.074308 (3.948880) | 11.122602 / 10.191392 (0.931210) | 0.141002 / 0.680424 (-0.539422) | 0.015955 / 0.534201 (-0.518246) | 0.270293 / 0.579283 (-0.308990) | 0.281839 / 0.434364 (-0.152525) | 0.307279 / 0.540337 (-0.233058) | 0.434687 / 1.386936 (-0.952249) |

</details>
</details>

![](https://cml.dev/watermark.png#7eaad71464e85c7358eaa36494227a43257ffcd8 "CML watermark")


What would be the impact for non-windows users ?

Also I wonder if a gc.collect() after the `del` could help to remove the PermissionError ? Or register the dataset for deletion on copy/pickle maybe ?

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004973 / 0.011353 (-0.006380) | 0.002753 / 0.011008 (-0.008256) | 0.061489 / 0.038508 (0.022981) | 0.051122 / 0.023109 (0.028012) | 0.228783 / 0.275898 (-0.047115) | 0.256982 / 0.323480 (-0.066498) | 0.002873 / 0.007986 (-0.005112) | 0.003544 / 0.004328 (-0.000784) | 0.048721 / 0.004250 (0.044471) | 0.039137 / 0.037052 (0.002085) | 0.244988 / 0.258489 (-0.013501) | 0.275230 / 0.293841 (-0.018611) | 0.023034 / 0.128546 (-0.105513) | 0.006988 / 0.075646 (-0.068658) | 0.202780 / 0.419271 (-0.216492) | 0.035325 / 0.043533 (-0.008207) | 0.241722 / 0.255139 (-0.013417) | 0.259671 / 0.283200 (-0.023528) | 0.019875 / 0.141683 (-0.121808) | 1.098667 / 1.452155 (-0.353488) | 1.161444 / 1.492716 (-0.331272) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093591 / 0.018006 (0.075585) | 0.298703 / 0.000490 (0.298213) | 0.000219 / 0.000200 (0.000019) | 0.000043 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018319 / 0.037411 (-0.019092) | 0.062993 / 0.014526 (0.048467) | 0.074313 / 0.176557 (-0.102244) | 0.123089 / 0.737135 (-0.614046) | 0.075177 / 0.296338 (-0.221162) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.268584 / 0.215209 (0.053375) | 2.633116 / 2.077655 (0.555461) | 1.390743 / 1.504120 (-0.113377) | 1.277385 / 1.541195 (-0.263810) | 1.287934 / 1.468490 (-0.180556) | 0.387934 / 4.584777 (-4.196843) | 2.345819 / 3.745712 (-1.399893) | 2.558169 / 5.269862 (-2.711693) | 1.569812 / 4.565676 (-2.995865) | 0.045297 / 0.424275 (-0.378978) | 0.005238 / 0.007607 (-0.002369) | 0.359704 / 0.226044 (0.133659) | 3.204688 / 2.268929 (0.935759) | 1.753321 / 55.444624 (-53.691303) | 1.492223 / 6.876477 (-5.384254) | 1.498207 / 2.142072 (-0.643865) | 0.459830 / 4.805227 (-4.345397) | 0.098194 / 6.500664 (-6.402470) | 0.042632 / 0.075469 (-0.032837) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.963020 / 1.841788 (-0.878768) | 11.500470 / 8.074308 (3.426161) | 10.451882 / 10.191392 (0.260490) | 0.127706 / 0.680424 (-0.552718) | 0.014084 / 0.534201 (-0.520117) | 0.269728 / 0.579283 (-0.309555) | 0.260283 / 0.434364 (-0.174080) | 0.303717 / 0.540337 (-0.236620) | 0.397028 / 1.386936 (-0.989908) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004823 / 0.011353 (-0.006529) | 0.002751 / 0.011008 (-0.008257) | 0.048719 / 0.038508 (0.010211) | 0.051409 / 0.023109 (0.028300) | 0.267139 / 0.275898 (-0.008759) | 0.287659 / 0.323480 (-0.035821) | 0.003959 / 0.007986 (-0.004027) | 0.002376 / 0.004328 (-0.001953) | 0.047942 / 0.004250 (0.043692) | 0.039742 / 0.037052 (0.002690) | 0.268348 / 0.258489 (0.009859) | 0.297201 / 0.293841 (0.003360) | 0.024226 / 0.128546 (-0.104320) | 0.007103 / 0.075646 (-0.068544) | 0.053310 / 0.419271 (-0.365961) | 0.032716 / 0.043533 (-0.010816) | 0.269469 / 0.255139 (0.014330) | 0.287752 / 0.283200 (0.004553) | 0.018191 / 0.141683 (-0.123492) | 1.114086 / 1.452155 (-0.338069) | 1.188054 / 1.492716 (-0.304662) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091072 / 0.018006 (0.073066) | 0.300367 / 0.000490 (0.299877) | 0.000218 / 0.000200 (0.000018) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.020970 / 0.037411 (-0.016441) | 0.070356 / 0.014526 (0.055830) | 0.081339 / 0.176557 (-0.095218) | 0.120741 / 0.737135 (-0.616394) | 0.081677 / 0.296338 (-0.214662) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.290405 / 0.215209 (0.075196) | 2.863877 / 2.077655 (0.786222) | 1.524603 / 1.504120 (0.020483) | 1.397917 / 1.541195 (-0.143278) | 1.402635 / 1.468490 (-0.065855) | 0.405525 / 4.584777 (-4.179252) | 2.432474 / 3.745712 (-1.313239) | 2.446277 / 5.269862 (-2.823585) | 1.550300 / 4.565676 (-3.015377) | 0.046545 / 0.424275 (-0.377730) | 0.004824 / 0.007607 (-0.002783) | 0.343578 / 0.226044 (0.117534) | 3.436850 / 2.268929 (1.167922) | 1.897200 / 55.444624 (-53.547425) | 1.625222 / 6.876477 (-5.251255) | 1.730488 / 2.142072 (-0.411585) | 0.482099 / 4.805227 (-4.323129) | 0.097828 / 6.500664 (-6.402836) | 0.040385 / 0.075469 (-0.035084) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.950975 / 1.841788 (-0.890812) | 11.875024 / 8.074308 (3.800715) | 10.430301 / 10.191392 (0.238909) | 0.130546 / 0.680424 (-0.549878) | 0.015423 / 0.534201 (-0.518778) | 0.269592 / 0.579283 (-0.309691) | 0.282505 / 0.434364 (-0.151859) | 0.305567 / 0.540337 (-0.234771) | 0.522142 / 1.386936 (-0.864794) |

</details>
</details>

![](https://cml.dev/watermark.png#c166692aa955528180dd4d55474a984f6044896d "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004983 / 0.011353 (-0.006369) | 0.003346 / 0.011008 (-0.007662) | 0.062233 / 0.038508 (0.023725) | 0.050246 / 0.023109 (0.027137) | 0.305738 / 0.275898 (0.029839) | 0.321863 / 0.323480 (-0.001617) | 0.003870 / 0.007986 (-0.004116) | 0.002610 / 0.004328 (-0.001718) | 0.047734 / 0.004250 (0.043483) | 0.037611 / 0.037052 (0.000559) | 0.299121 / 0.258489 (0.040632) | 0.327370 / 0.293841 (0.033529) | 0.027009 / 0.128546 (-0.101537) | 0.010816 / 0.075646 (-0.064830) | 0.204627 / 0.419271 (-0.214645) | 0.035708 / 0.043533 (-0.007825) | 0.291837 / 0.255139 (0.036698) | 0.313646 / 0.283200 (0.030447) | 0.017277 / 0.141683 (-0.124405) | 1.097907 / 1.452155 (-0.354248) | 1.163203 / 1.492716 (-0.329513) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091933 / 0.018006 (0.073926) | 0.298787 / 0.000490 (0.298297) | 0.000204 / 0.000200 (0.000004) | 0.000051 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018349 / 0.037411 (-0.019062) | 0.061520 / 0.014526 (0.046994) | 0.073159 / 0.176557 (-0.103397) | 0.118657 / 0.737135 (-0.618478) | 0.073601 / 0.296338 (-0.222737) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.276297 / 0.215209 (0.061088) | 2.725668 / 2.077655 (0.648013) | 1.458079 / 1.504120 (-0.046041) | 1.331236 / 1.541195 (-0.209959) | 1.347919 / 1.468490 (-0.120571) | 0.565954 / 4.584777 (-4.018823) | 2.380883 / 3.745712 (-1.364829) | 2.800533 / 5.269862 (-2.469329) | 1.740534 / 4.565676 (-2.825142) | 0.065617 / 0.424275 (-0.358658) | 0.004907 / 0.007607 (-0.002700) | 0.335973 / 0.226044 (0.109929) | 3.337405 / 2.268929 (1.068476) | 1.819852 / 55.444624 (-53.624772) | 1.542724 / 6.876477 (-5.333752) | 1.509508 / 2.142072 (-0.632565) | 0.648618 / 4.805227 (-4.156609) | 0.116812 / 6.500664 (-6.383852) | 0.041561 / 0.075469 (-0.033909) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.943488 / 1.841788 (-0.898299) | 11.184770 / 8.074308 (3.110462) | 10.406311 / 10.191392 (0.214919) | 0.129841 / 0.680424 (-0.550583) | 0.013736 / 0.534201 (-0.520465) | 0.287281 / 0.579283 (-0.292002) | 0.267403 / 0.434364 (-0.166961) | 0.325319 / 0.540337 (-0.215019) | 0.454207 / 1.386936 (-0.932729) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005169 / 0.011353 (-0.006183) | 0.003155 / 0.011008 (-0.007854) | 0.048101 / 0.038508 (0.009593) | 0.048726 / 0.023109 (0.025617) | 0.275768 / 0.275898 (-0.000130) | 0.291209 / 0.323480 (-0.032271) | 0.003984 / 0.007986 (-0.004001) | 0.002586 / 0.004328 (-0.001742) | 0.047751 / 0.004250 (0.043500) | 0.040176 / 0.037052 (0.003124) | 0.279161 / 0.258489 (0.020672) | 0.297371 / 0.293841 (0.003530) | 0.028502 / 0.128546 (-0.100044) | 0.010103 / 0.075646 (-0.065544) | 0.056920 / 0.419271 (-0.362351) | 0.032174 / 0.043533 (-0.011359) | 0.271925 / 0.255139 (0.016786) | 0.289572 / 0.283200 (0.006372) | 0.017981 / 0.141683 (-0.123702) | 1.192972 / 1.452155 (-0.259183) | 1.223231 / 1.492716 (-0.269485) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091363 / 0.018006 (0.073356) | 0.298106 / 0.000490 (0.297616) | 0.000216 / 0.000200 (0.000016) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021509 / 0.037411 (-0.015902) | 0.068377 / 0.014526 (0.053851) | 0.079798 / 0.176557 (-0.096759) | 0.120546 / 0.737135 (-0.616589) | 0.080602 / 0.296338 (-0.215737) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.300809 / 0.215209 (0.085600) | 2.921144 / 2.077655 (0.843489) | 1.621096 / 1.504120 (0.116976) | 1.504265 / 1.541195 (-0.036930) | 1.508050 / 1.468490 (0.039560) | 0.554291 / 4.584777 (-4.030486) | 2.418798 / 3.745712 (-1.326914) | 2.768088 / 5.269862 (-2.501773) | 1.728267 / 4.565676 (-2.837410) | 0.062943 / 0.424275 (-0.361332) | 0.004891 / 0.007607 (-0.002716) | 0.350298 / 0.226044 (0.124254) | 3.442782 / 2.268929 (1.173853) | 1.960163 / 55.444624 (-53.484461) | 1.682000 / 6.876477 (-5.194477) | 1.680311 / 2.142072 (-0.461761) | 0.631201 / 4.805227 (-4.174026) | 0.115211 / 6.500664 (-6.385453) | 0.041279 / 0.075469 (-0.034190) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.962478 / 1.841788 (-0.879310) | 11.671463 / 8.074308 (3.597155) | 10.640129 / 10.191392 (0.448737) | 0.130649 / 0.680424 (-0.549775) | 0.016169 / 0.534201 (-0.518032) | 0.286894 / 0.579283 (-0.292389) | 0.269319 / 0.434364 (-0.165045) | 0.324512 / 0.540337 (-0.215825) | 0.550874 / 1.386936 (-0.836062) |

</details>
</details>

![](https://cml.dev/watermark.png#69f135121beb1616f1d7c7584b317d4e41e21275 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005078 / 0.011353 (-0.006275) | 0.003950 / 0.011008 (-0.007058) | 0.063345 / 0.038508 (0.024837) | 0.054486 / 0.023109 (0.031377) | 0.243213 / 0.275898 (-0.032685) | 0.264079 / 0.323480 (-0.059401) | 0.003922 / 0.007986 (-0.004064) | 0.002631 / 0.004328 (-0.001698) | 0.048660 / 0.004250 (0.044409) | 0.037205 / 0.037052 (0.000153) | 0.244577 / 0.258489 (-0.013912) | 0.276025 / 0.293841 (-0.017816) | 0.027134 / 0.128546 (-0.101412) | 0.010921 / 0.075646 (-0.064726) | 0.209792 / 0.419271 (-0.209479) | 0.035999 / 0.043533 (-0.007534) | 0.245671 / 0.255139 (-0.009468) | 0.262807 / 0.283200 (-0.020393) | 0.018173 / 0.141683 (-0.123510) | 1.084417 / 1.452155 (-0.367738) | 1.148284 / 1.492716 (-0.344432) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093128 / 0.018006 (0.075122) | 0.301606 / 0.000490 (0.301117) | 0.000221 / 0.000200 (0.000021) | 0.000050 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018718 / 0.037411 (-0.018693) | 0.060819 / 0.014526 (0.046293) | 0.073050 / 0.176557 (-0.103507) | 0.120043 / 0.737135 (-0.617092) | 0.075374 / 0.296338 (-0.220965) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.291080 / 0.215209 (0.075871) | 2.808802 / 2.077655 (0.731148) | 1.485686 / 1.504120 (-0.018434) | 1.354356 / 1.541195 (-0.186839) | 1.347863 / 1.468490 (-0.120627) | 0.571501 / 4.584777 (-4.013276) | 2.377960 / 3.745712 (-1.367752) | 2.768023 / 5.269862 (-2.501839) | 1.754360 / 4.565676 (-2.811316) | 0.063115 / 0.424275 (-0.361160) | 0.004941 / 0.007607 (-0.002666) | 0.338281 / 0.226044 (0.112237) | 3.340587 / 2.268929 (1.071658) | 1.849479 / 55.444624 (-53.595145) | 1.551846 / 6.876477 (-5.324631) | 1.539090 / 2.142072 (-0.602983) | 0.644522 / 4.805227 (-4.160705) | 0.117398 / 6.500664 (-6.383266) | 0.042239 / 0.075469 (-0.033230) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.949496 / 1.841788 (-0.892291) | 11.548352 / 8.074308 (3.474044) | 10.478065 / 10.191392 (0.286673) | 0.129534 / 0.680424 (-0.550890) | 0.015378 / 0.534201 (-0.518822) | 0.287221 / 0.579283 (-0.292062) | 0.262944 / 0.434364 (-0.171419) | 0.321727 / 0.540337 (-0.218611) | 0.432354 / 1.386936 (-0.954582) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005256 / 0.011353 (-0.006097) | 0.003491 / 0.011008 (-0.007517) | 0.048647 / 0.038508 (0.010139) | 0.054011 / 0.023109 (0.030901) | 0.271786 / 0.275898 (-0.004112) | 0.291964 / 0.323480 (-0.031516) | 0.004035 / 0.007986 (-0.003950) | 0.002671 / 0.004328 (-0.001657) | 0.048108 / 0.004250 (0.043857) | 0.040421 / 0.037052 (0.003368) | 0.278594 / 0.258489 (0.020105) | 0.300707 / 0.293841 (0.006867) | 0.028924 / 0.128546 (-0.099623) | 0.010600 / 0.075646 (-0.065047) | 0.057649 / 0.419271 (-0.361623) | 0.034221 / 0.043533 (-0.009312) | 0.276692 / 0.255139 (0.021553) | 0.293545 / 0.283200 (0.010345) | 0.017908 / 0.141683 (-0.123775) | 1.135108 / 1.452155 (-0.317047) | 1.190823 / 1.492716 (-0.301893) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095243 / 0.018006 (0.077237) | 0.301885 / 0.000490 (0.301396) | 0.000235 / 0.000200 (0.000035) | 0.000056 / 0.000054 (0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021561 / 0.037411 (-0.015850) | 0.069054 / 0.014526 (0.054529) | 0.080466 / 0.176557 (-0.096091) | 0.121323 / 0.737135 (-0.615812) | 0.081891 / 0.296338 (-0.214448) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.293957 / 0.215209 (0.078748) | 2.869035 / 2.077655 (0.791380) | 1.608837 / 1.504120 (0.104717) | 1.440594 / 1.541195 (-0.100601) | 1.464775 / 1.468490 (-0.003715) | 0.565663 / 4.584777 (-4.019114) | 2.439456 / 3.745712 (-1.306256) | 2.794775 / 5.269862 (-2.475087) | 1.750026 / 4.565676 (-2.815651) | 0.063291 / 0.424275 (-0.360984) | 0.004930 / 0.007607 (-0.002677) | 0.347169 / 0.226044 (0.121125) | 3.408260 / 2.268929 (1.139331) | 1.920933 / 55.444624 (-53.523691) | 1.648821 / 6.876477 (-5.227656) | 1.639022 / 2.142072 (-0.503051) | 0.642870 / 4.805227 (-4.162357) | 0.117077 / 6.500664 (-6.383587) | 0.040784 / 0.075469 (-0.034685) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.993501 / 1.841788 (-0.848287) | 12.012423 / 8.074308 (3.938115) | 10.740932 / 10.191392 (0.549540) | 0.132409 / 0.680424 (-0.548015) | 0.015294 / 0.534201 (-0.518907) | 0.287902 / 0.579283 (-0.291381) | 0.281350 / 0.434364 (-0.153014) | 0.329201 / 0.540337 (-0.211137) | 0.553199 / 1.386936 (-0.833737) |

</details>
</details>

![](https://cml.dev/watermark.png#ecd3a22c5dec2133491a320515e12956512439eb "CML watermark")