Simplify filesystem logic
State: closed | Author: mariosasko (COLLABORATOR)

Simplifies the existing filesystem logic (e.g., to avoid unnecessary if-else as mentioned in https://github.com/huggingface/datasets/pull/6098#issue-1827655071)

Comments: <details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008852 / 0.011353 (-0.002501) | 0.004613 / 0.011008 (-0.006396) | 0.096153 / 0.038508 (0.057645) | 0.074945 / 0.023109 (0.051836) | 0.365960 / 0.275898 (0.090062) | 0.385450 / 0.323480 (0.061970) | 0.004757 / 0.007986 (-0.003229) | 0.003453 / 0.004328 (-0.000876) | 0.069944 / 0.004250 (0.065693) | 0.057781 / 0.037052 (0.020729) | 0.361056 / 0.258489 (0.102567) | 0.409218 / 0.293841 (0.115377) | 0.045714 / 0.128546 (-0.082833) | 0.013776 / 0.075646 (-0.061871) | 0.328797 / 0.419271 (-0.090474) | 0.063431 / 0.043533 (0.019899) | 0.370799 / 0.255139 (0.115660) | 0.370701 / 0.283200 (0.087502) | 0.034894 / 0.141683 (-0.106789) | 1.730290 / 1.452155 (0.278136) | 1.863600 / 1.492716 (0.370883) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.245571 / 0.018006 (0.227565) | 0.509666 / 0.000490 (0.509176) | 0.008051 / 0.000200 (0.007851) | 0.000104 / 0.000054 (0.000050) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027854 / 0.037411 (-0.009557) | 0.090735 / 0.014526 (0.076209) | 0.100100 / 0.176557 (-0.076457) | 0.158267 / 0.737135 (-0.578868) | 0.107537 / 0.296338 (-0.188801) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.565455 / 0.215209 (0.350246) | 5.671436 / 2.077655 (3.593781) | 2.438078 / 1.504120 (0.933958) | 2.072403 / 1.541195 (0.531208) | 2.127830 / 1.468490 (0.659340) | 0.840101 / 4.584777 (-3.744675) | 4.945952 / 3.745712 (1.200240) | 4.840904 / 5.269862 (-0.428957) | 3.037936 / 4.565676 (-1.527740) | 0.099027 / 0.424275 (-0.325248) | 0.008448 / 0.007607 (0.000841) | 0.703315 / 0.226044 (0.477271) | 6.837550 / 2.268929 (4.568621) | 3.204232 / 55.444624 (-52.240393) | 2.492985 / 6.876477 (-4.383492) | 2.426792 / 2.142072 (0.284720) | 0.998430 / 4.805227 (-3.806797) | 0.203854 / 6.500664 (-6.296811) | 0.072386 / 0.075469 (-0.003083) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.606627 / 1.841788 (-0.235161) | 22.287391 / 8.074308 (14.213082) | 20.245654 / 10.191392 (10.054262) | 0.229377 / 0.680424 (-0.451046) | 0.028399 / 0.534201 (-0.505802) | 0.446567 / 0.579283 (-0.132716) | 0.565277 / 0.434364 (0.130913) | 0.502957 / 0.540337 (-0.037381) | 0.749268 / 1.386936 (-0.637668) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008253 / 0.011353 (-0.003100) | 0.004432 / 0.011008 (-0.006576) | 0.081995 / 0.038508 (0.043487) | 0.075443 / 0.023109 (0.052334) | 0.442139 / 0.275898 (0.166241) | 0.507308 / 0.323480 (0.183829) | 0.007343 / 0.007986 (-0.000643) | 0.003850 / 0.004328 (-0.000478) | 0.072656 / 0.004250 (0.068406) | 0.054585 / 0.037052 (0.017533) | 0.430057 / 0.258489 (0.171568) | 0.466953 / 0.293841 (0.173112) | 0.050350 / 0.128546 (-0.078196) | 0.013682 / 0.075646 (-0.061965) | 0.088164 / 0.419271 (-0.331107) | 0.061726 / 0.043533 (0.018193) | 0.444420 / 0.255139 (0.189281) | 0.470406 / 0.283200 (0.187206) | 0.033258 / 0.141683 (-0.108425) | 1.635977 / 1.452155 (0.183823) | 1.732767 / 1.492716 (0.240051) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.227350 / 0.018006 (0.209344) | 0.500805 / 0.000490 (0.500316) | 0.006473 / 0.000200 (0.006273) | 0.000110 / 0.000054 (0.000055) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034456 / 0.037411 (-0.002955) | 0.094832 / 0.014526 (0.080306) | 0.118549 / 0.176557 (-0.058008) | 0.177971 / 0.737135 (-0.559164) | 0.114165 / 0.296338 (-0.182174) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.664805 / 0.215209 (0.449596) | 6.509756 / 2.077655 (4.432101) | 2.936840 / 1.504120 (1.432720) | 2.662645 / 1.541195 (1.121450) | 2.659957 / 1.468490 (1.191467) | 0.903019 / 4.584777 (-3.681758) | 5.237191 / 3.745712 (1.491479) | 4.791917 / 5.269862 (-0.477945) | 3.130905 / 4.565676 (-1.434772) | 0.100953 / 0.424275 (-0.323322) | 0.008388 / 0.007607 (0.000781) | 0.776393 / 0.226044 (0.550348) | 7.726230 / 2.268929 (5.457301) | 3.669223 / 55.444624 (-51.775401) | 2.904556 / 6.876477 (-3.971921) | 3.205546 / 2.142072 (1.063473) | 1.058899 / 4.805227 (-3.746329) | 0.213733 / 6.500664 (-6.286931) | 0.071374 / 0.075469 (-0.004096) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.713384 / 1.841788 (-0.128403) | 23.325498 / 8.074308 (15.251190) | 20.140510 / 10.191392 (9.949118) | 0.211565 / 0.680424 (-0.468859) | 0.032916 / 0.534201 (-0.501285) | 0.460504 / 0.579283 (-0.118779) | 0.594352 / 0.434364 (0.159988) | 0.556384 / 0.540337 (0.016047) | 0.788586 / 1.386936 (-0.598350) |

</details>
</details>

![](https://cml.dev/watermark.png#3c7249c11d8330ce49b1fe119c34fc6100f10774 "CML watermark")


_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008840 / 0.011353 (-0.002513) | 0.005045 / 0.011008 (-0.005963) | 0.110777 / 0.038508 (0.072269) | 0.100495 / 0.023109 (0.077386) | 0.420302 / 0.275898 (0.144404) | 0.456423 / 0.323480 (0.132943) | 0.006873 / 0.007986 (-0.001113) | 0.005230 / 0.004328 (0.000902) | 0.081316 / 0.004250 (0.077066) | 0.063047 / 0.037052 (0.025995) | 0.439469 / 0.258489 (0.180979) | 0.488477 / 0.293841 (0.194636) | 0.048553 / 0.128546 (-0.079994) | 0.014984 / 0.075646 (-0.060662) | 0.401317 / 0.419271 (-0.017955) | 0.074578 / 0.043533 (0.031045) | 0.435298 / 0.255139 (0.180159) | 0.464406 / 0.283200 (0.181206) | 0.048788 / 0.141683 (-0.092895) | 1.836166 / 1.452155 (0.384011) | 1.959808 / 1.492716 (0.467091) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.321419 / 0.018006 (0.303412) | 0.595736 / 0.000490 (0.595246) | 0.021144 / 0.000200 (0.020944) | 0.000626 / 0.000054 (0.000571) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033033 / 0.037411 (-0.004379) | 0.112621 / 0.014526 (0.098095) | 0.118736 / 0.176557 (-0.057821) | 0.195533 / 0.737135 (-0.541602) | 0.120807 / 0.296338 (-0.175531) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.616692 / 0.215209 (0.401483) | 6.033674 / 2.077655 (3.956019) | 2.630106 / 1.504120 (1.125986) | 2.316739 / 1.541195 (0.775544) | 2.387525 / 1.468490 (0.919035) | 0.863385 / 4.584777 (-3.721392) | 5.288193 / 3.745712 (1.542481) | 5.115766 / 5.269862 (-0.154096) | 3.083055 / 4.565676 (-1.482621) | 0.104885 / 0.424275 (-0.319391) | 0.012233 / 0.007607 (0.004626) | 0.739924 / 0.226044 (0.513880) | 7.422996 / 2.268929 (5.154067) | 3.403316 / 55.444624 (-52.041309) | 2.778740 / 6.876477 (-4.097736) | 2.836937 / 2.142072 (0.694864) | 1.059683 / 4.805227 (-3.745544) | 0.235838 / 6.500664 (-6.264826) | 0.083725 / 0.075469 (0.008256) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.755843 / 1.841788 (-0.085944) | 25.186642 / 8.074308 (17.112334) | 24.133582 / 10.191392 (13.942190) | 0.240511 / 0.680424 (-0.439913) | 0.029563 / 0.534201 (-0.504638) | 0.486049 / 0.579283 (-0.093234) | 0.610064 / 0.434364 (0.175700) | 0.559521 / 0.540337 (0.019184) | 0.828289 / 1.386936 (-0.558647) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.012134 / 0.011353 (0.000781) | 0.005133 / 0.011008 (-0.005875) | 0.084521 / 0.038508 (0.046013) | 0.095172 / 0.023109 (0.072063) | 0.527298 / 0.275898 (0.251400) | 0.558915 / 0.323480 (0.235435) | 0.006996 / 0.007986 (-0.000989) | 0.004283 / 0.004328 (-0.000045) | 0.082975 / 0.004250 (0.078725) | 0.067976 / 0.037052 (0.030924) | 0.534020 / 0.258489 (0.275531) | 0.560810 / 0.293841 (0.266969) | 0.051603 / 0.128546 (-0.076943) | 0.013330 / 0.075646 (-0.062316) | 0.094093 / 0.419271 (-0.325178) | 0.068967 / 0.043533 (0.025434) | 0.512527 / 0.255139 (0.257388) | 0.542182 / 0.283200 (0.258982) | 0.039159 / 0.141683 (-0.102524) | 1.858841 / 1.452155 (0.406686) | 1.915450 / 1.492716 (0.422734) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.269013 / 0.018006 (0.251007) | 0.601711 / 0.000490 (0.601222) | 0.013950 / 0.000200 (0.013750) | 0.000166 / 0.000054 (0.000112) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.038817 / 0.037411 (0.001405) | 0.138528 / 0.014526 (0.124002) | 0.130691 / 0.176557 (-0.045865) | 0.192825 / 0.737135 (-0.544310) | 0.128337 / 0.296338 (-0.168002) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.678725 / 0.215209 (0.463516) | 6.869763 / 2.077655 (4.792108) | 3.416224 / 1.504120 (1.912104) | 3.106971 / 1.541195 (1.565776) | 3.117248 / 1.468490 (1.648757) | 0.895004 / 4.584777 (-3.689773) | 5.551618 / 3.745712 (1.805906) | 4.964811 / 5.269862 (-0.305051) | 3.239555 / 4.565676 (-1.326121) | 0.099776 / 0.424275 (-0.324500) | 0.008723 / 0.007607 (0.001116) | 0.818554 / 0.226044 (0.592510) | 8.015976 / 2.268929 (5.747047) | 4.200392 / 55.444624 (-51.244232) | 3.566942 / 6.876477 (-3.309535) | 3.766249 / 2.142072 (1.624177) | 1.083428 / 4.805227 (-3.721799) | 0.214614 / 6.500664 (-6.286050) | 0.081951 / 0.075469 (0.006482) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.854400 / 1.841788 (0.012612) | 26.002556 / 8.074308 (17.928248) | 24.315194 / 10.191392 (14.123802) | 0.249012 / 0.680424 (-0.431412) | 0.032681 / 0.534201 (-0.501520) | 0.502360 / 0.579283 (-0.076923) | 0.606014 / 0.434364 (0.171650) | 0.616852 / 0.540337 (0.076514) | 0.861785 / 1.386936 (-0.525151) |

</details>
</details>

![](https://cml.dev/watermark.png#53c5c01583153cc112a507082aff4679433a1cce "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006723 / 0.011353 (-0.004630) | 0.004135 / 0.011008 (-0.006873) | 0.079241 / 0.038508 (0.040733) | 0.065484 / 0.023109 (0.042374) | 0.302831 / 0.275898 (0.026933) | 0.343747 / 0.323480 (0.020268) | 0.005910 / 0.007986 (-0.002076) | 0.006028 / 0.004328 (0.001699) | 0.064000 / 0.004250 (0.059750) | 0.047872 / 0.037052 (0.010820) | 0.336928 / 0.258489 (0.078439) | 0.357726 / 0.293841 (0.063885) | 0.039375 / 0.128546 (-0.089171) | 0.010439 / 0.075646 (-0.065207) | 0.310453 / 0.419271 (-0.108819) | 0.055320 / 0.043533 (0.011787) | 0.294722 / 0.255139 (0.039583) | 0.314649 / 0.283200 (0.031450) | 0.033223 / 0.141683 (-0.108460) | 1.386705 / 1.452155 (-0.065450) | 1.420546 / 1.492716 (-0.072170) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.262649 / 0.018006 (0.244643) | 0.536764 / 0.000490 (0.536274) | 0.011090 / 0.000200 (0.010891) | 0.000118 / 0.000054 (0.000063) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023822 / 0.037411 (-0.013590) | 0.074279 / 0.014526 (0.059753) | 0.081295 / 0.176557 (-0.095262) | 0.135853 / 0.737135 (-0.601282) | 0.080193 / 0.296338 (-0.216146) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.468577 / 0.215209 (0.253368) | 4.615975 / 2.077655 (2.538321) | 2.059232 / 1.504120 (0.555112) | 1.798578 / 1.541195 (0.257383) | 1.801436 / 1.468490 (0.332946) | 0.660489 / 4.584777 (-3.924288) | 4.394652 / 3.745712 (0.648940) | 3.956277 / 5.269862 (-1.313585) | 2.406700 / 4.565676 (-2.158976) | 0.077174 / 0.424275 (-0.347101) | 0.007121 / 0.007607 (-0.000486) | 0.568213 / 0.226044 (0.342168) | 5.721217 / 2.268929 (3.452289) | 2.662741 / 55.444624 (-52.781883) | 2.207333 / 6.876477 (-4.669144) | 2.165279 / 2.142072 (0.023206) | 0.772566 / 4.805227 (-4.032661) | 0.162845 / 6.500664 (-6.337819) | 0.057515 / 0.075469 (-0.017954) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.313565 / 1.841788 (-0.528223) | 19.298926 / 8.074308 (11.224618) | 17.194320 / 10.191392 (7.002928) | 0.223404 / 0.680424 (-0.457020) | 0.024735 / 0.534201 (-0.509466) | 0.388452 / 0.579283 (-0.190831) | 0.489354 / 0.434364 (0.054990) | 0.427962 / 0.540337 (-0.112375) | 0.629483 / 1.386936 (-0.757453) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007404 / 0.011353 (-0.003949) | 0.004434 / 0.011008 (-0.006574) | 0.061633 / 0.038508 (0.023125) | 0.058446 / 0.023109 (0.035336) | 0.386107 / 0.275898 (0.110209) | 0.397676 / 0.323480 (0.074197) | 0.005463 / 0.007986 (-0.002523) | 0.003797 / 0.004328 (-0.000531) | 0.067323 / 0.004250 (0.063072) | 0.053826 / 0.037052 (0.016774) | 0.387910 / 0.258489 (0.129421) | 0.409364 / 0.293841 (0.115523) | 0.039836 / 0.128546 (-0.088710) | 0.011940 / 0.075646 (-0.063706) | 0.071812 / 0.419271 (-0.347459) | 0.047952 / 0.043533 (0.004419) | 0.386826 / 0.255139 (0.131687) | 0.392845 / 0.283200 (0.109645) | 0.029430 / 0.141683 (-0.112253) | 1.390961 / 1.452155 (-0.061194) | 1.482744 / 1.492716 (-0.009972) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.258814 / 0.018006 (0.240807) | 0.535505 / 0.000490 (0.535015) | 0.006097 / 0.000200 (0.005897) | 0.000130 / 0.000054 (0.000075) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028046 / 0.037411 (-0.009365) | 0.078077 / 0.014526 (0.063552) | 0.087713 / 0.176557 (-0.088843) | 0.140856 / 0.737135 (-0.596279) | 0.090565 / 0.296338 (-0.205773) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.504375 / 0.215209 (0.289165) | 5.133472 / 2.077655 (3.055817) | 2.368968 / 1.504120 (0.864848) | 2.176939 / 1.541195 (0.635744) | 2.151976 / 1.468490 (0.683486) | 0.720566 / 4.584777 (-3.864211) | 5.050505 / 3.745712 (1.304793) | 3.993614 / 5.269862 (-1.276248) | 2.492234 / 4.565676 (-2.073443) | 0.089629 / 0.424275 (-0.334646) | 0.008074 / 0.007607 (0.000467) | 0.677706 / 0.226044 (0.451661) | 6.208332 / 2.268929 (3.939403) | 3.058299 / 55.444624 (-52.386325) | 2.461078 / 6.876477 (-4.415399) | 2.622681 / 2.142072 (0.480609) | 0.873573 / 4.805227 (-3.931654) | 0.176321 / 6.500664 (-6.324343) | 0.062410 / 0.075469 (-0.013059) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.454767 / 1.841788 (-0.387021) | 19.544225 / 8.074308 (11.469917) | 17.365997 / 10.191392 (7.174605) | 0.225461 / 0.680424 (-0.454963) | 0.027679 / 0.534201 (-0.506522) | 0.396419 / 0.579283 (-0.182864) | 0.513244 / 0.434364 (0.078880) | 0.469054 / 0.540337 (-0.071283) | 0.676458 / 1.386936 (-0.710478) |

</details>
</details>

![](https://cml.dev/watermark.png#d44d8a649b541cd0b10ea99fbfe7a02c3ba50a63 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007606 / 0.011353 (-0.003747) | 0.004692 / 0.011008 (-0.006317) | 0.100525 / 0.038508 (0.062017) | 0.085426 / 0.023109 (0.062317) | 0.378568 / 0.275898 (0.102670) | 0.412268 / 0.323480 (0.088788) | 0.004756 / 0.007986 (-0.003230) | 0.003871 / 0.004328 (-0.000457) | 0.075244 / 0.004250 (0.070994) | 0.064969 / 0.037052 (0.027916) | 0.385569 / 0.258489 (0.127079) | 0.429117 / 0.293841 (0.135276) | 0.035798 / 0.128546 (-0.092749) | 0.009999 / 0.075646 (-0.065647) | 0.351380 / 0.419271 (-0.067891) | 0.060850 / 0.043533 (0.017317) | 0.381327 / 0.255139 (0.126188) | 0.403663 / 0.283200 (0.120464) | 0.028103 / 0.141683 (-0.113580) | 1.814143 / 1.452155 (0.361988) | 1.895062 / 1.492716 (0.402346) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.263581 / 0.018006 (0.245575) | 0.506988 / 0.000490 (0.506499) | 0.012775 / 0.000200 (0.012575) | 0.000456 / 0.000054 (0.000402) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033452 / 0.037411 (-0.003959) | 0.104950 / 0.014526 (0.090425) | 0.114803 / 0.176557 (-0.061754) | 0.182465 / 0.737135 (-0.554671) | 0.116156 / 0.296338 (-0.180183) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.441574 / 0.215209 (0.226365) | 4.394601 / 2.077655 (2.316946) | 2.170797 / 1.504120 (0.666677) | 1.926675 / 1.541195 (0.385480) | 1.974867 / 1.468490 (0.506377) | 0.546777 / 4.584777 (-4.038000) | 4.053612 / 3.745712 (0.307900) | 3.934278 / 5.269862 (-1.335583) | 2.354660 / 4.565676 (-2.211017) | 0.067706 / 0.424275 (-0.356569) | 0.009217 / 0.007607 (0.001610) | 0.539261 / 0.226044 (0.313217) | 5.409552 / 2.268929 (3.140623) | 2.835739 / 55.444624 (-52.608886) | 2.282246 / 6.876477 (-4.594230) | 2.359930 / 2.142072 (0.217858) | 0.696363 / 4.805227 (-4.108864) | 0.155947 / 6.500664 (-6.344717) | 0.071293 / 0.075469 (-0.004176) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.495512 / 1.841788 (-0.346275) | 22.027128 / 8.074308 (13.952820) | 16.226068 / 10.191392 (6.034676) | 0.180281 / 0.680424 (-0.500142) | 0.021839 / 0.534201 (-0.512362) | 0.446151 / 0.579283 (-0.133132) | 0.476872 / 0.434364 (0.042508) | 0.515171 / 0.540337 (-0.025166) | 0.731372 / 1.386936 (-0.655564) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006843 / 0.011353 (-0.004510) | 0.004286 / 0.011008 (-0.006722) | 0.074104 / 0.038508 (0.035596) | 0.076789 / 0.023109 (0.053680) | 0.441506 / 0.275898 (0.165608) | 0.500999 / 0.323480 (0.177519) | 0.006041 / 0.007986 (-0.001945) | 0.003718 / 0.004328 (-0.000610) | 0.074189 / 0.004250 (0.069938) | 0.060513 / 0.037052 (0.023461) | 0.460812 / 0.258489 (0.202323) | 0.503631 / 0.293841 (0.209790) | 0.037026 / 0.128546 (-0.091520) | 0.009611 / 0.075646 (-0.066035) | 0.077037 / 0.419271 (-0.342234) | 0.052191 / 0.043533 (0.008658) | 0.444567 / 0.255139 (0.189428) | 0.486730 / 0.283200 (0.203530) | 0.023846 / 0.141683 (-0.117837) | 1.692422 / 1.452155 (0.240267) | 1.809648 / 1.492716 (0.316932) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.240007 / 0.018006 (0.222001) | 0.481980 / 0.000490 (0.481490) | 0.006945 / 0.000200 (0.006746) | 0.000120 / 0.000054 (0.000065) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.037198 / 0.037411 (-0.000213) | 0.119413 / 0.014526 (0.104887) | 0.137409 / 0.176557 (-0.039148) | 0.199130 / 0.737135 (-0.538005) | 0.133137 / 0.296338 (-0.163202) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.521747 / 0.215209 (0.306538) | 4.955653 / 2.077655 (2.877999) | 2.694323 / 1.504120 (1.190203) | 2.496629 / 1.541195 (0.955434) | 2.661151 / 1.468490 (1.192660) | 0.576687 / 4.584777 (-4.008089) | 4.251437 / 3.745712 (0.505725) | 3.683020 / 5.269862 (-1.586842) | 2.363951 / 4.565676 (-2.201726) | 0.064631 / 0.424275 (-0.359644) | 0.007958 / 0.007607 (0.000351) | 0.616498 / 0.226044 (0.390454) | 5.919424 / 2.268929 (3.650496) | 3.255936 / 55.444624 (-52.188689) | 2.866167 / 6.876477 (-4.010309) | 3.007272 / 2.142072 (0.865199) | 0.660259 / 4.805227 (-4.144968) | 0.152469 / 6.500664 (-6.348195) | 0.065254 / 0.075469 (-0.010215) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.547912 / 1.841788 (-0.293876) | 22.494611 / 8.074308 (14.420303) | 16.400746 / 10.191392 (6.209354) | 0.184137 / 0.680424 (-0.496287) | 0.023615 / 0.534201 (-0.510586) | 0.473923 / 0.579283 (-0.105360) | 0.473030 / 0.434364 (0.038666) | 0.534264 / 0.540337 (-0.006073) | 0.770178 / 1.386936 (-0.616758) |

</details>
</details>

![](https://cml.dev/watermark.png#565efb7f43839072ef01247681645ca404ba0b94 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006812 / 0.011353 (-0.004541) | 0.004254 / 0.011008 (-0.006754) | 0.084271 / 0.038508 (0.045763) | 0.084299 / 0.023109 (0.061189) | 0.317437 / 0.275898 (0.041539) | 0.350855 / 0.323480 (0.027375) | 0.004296 / 0.007986 (-0.003690) | 0.003610 / 0.004328 (-0.000718) | 0.065205 / 0.004250 (0.060955) | 0.057734 / 0.037052 (0.020682) | 0.324049 / 0.258489 (0.065560) | 0.365042 / 0.293841 (0.071201) | 0.031454 / 0.128546 (-0.097092) | 0.008703 / 0.075646 (-0.066943) | 0.286603 / 0.419271 (-0.132668) | 0.052251 / 0.043533 (0.008719) | 0.312404 / 0.255139 (0.057265) | 0.335902 / 0.283200 (0.052703) | 0.025087 / 0.141683 (-0.116595) | 1.478573 / 1.452155 (0.026418) | 1.559548 / 1.492716 (0.066831) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.307637 / 0.018006 (0.289631) | 0.567169 / 0.000490 (0.566679) | 0.006782 / 0.000200 (0.006582) | 0.000235 / 0.000054 (0.000180) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030979 / 0.037411 (-0.006433) | 0.089972 / 0.014526 (0.075446) | 0.101689 / 0.176557 (-0.074868) | 0.162038 / 0.737135 (-0.575097) | 0.103107 / 0.296338 (-0.193232) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.382458 / 0.215209 (0.167248) | 3.813105 / 2.077655 (1.735450) | 1.855198 / 1.504120 (0.351078) | 1.699850 / 1.541195 (0.158656) | 1.902818 / 1.468490 (0.434328) | 0.478654 / 4.584777 (-4.106123) | 3.536926 / 3.745712 (-0.208786) | 3.558557 / 5.269862 (-1.711304) | 2.121098 / 4.565676 (-2.444579) | 0.056584 / 0.424275 (-0.367691) | 0.007693 / 0.007607 (0.000086) | 0.471157 / 0.226044 (0.245112) | 4.717742 / 2.268929 (2.448813) | 2.389033 / 55.444624 (-53.055591) | 2.102898 / 6.876477 (-4.773579) | 2.233404 / 2.142072 (0.091332) | 0.585829 / 4.805227 (-4.219398) | 0.133784 / 6.500664 (-6.366880) | 0.063963 / 0.075469 (-0.011506) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.272234 / 1.841788 (-0.569554) | 19.897647 / 8.074308 (11.823339) | 14.808090 / 10.191392 (4.616698) | 0.167199 / 0.680424 (-0.513224) | 0.018357 / 0.534201 (-0.515844) | 0.391635 / 0.579283 (-0.187648) | 0.409603 / 0.434364 (-0.024761) | 0.467670 / 0.540337 (-0.072668) | 0.639763 / 1.386936 (-0.747173) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006794 / 0.011353 (-0.004559) | 0.004317 / 0.011008 (-0.006692) | 0.065434 / 0.038508 (0.026926) | 0.079066 / 0.023109 (0.055957) | 0.415486 / 0.275898 (0.139588) | 0.448072 / 0.323480 (0.124593) | 0.005705 / 0.007986 (-0.002281) | 0.003589 / 0.004328 (-0.000739) | 0.065195 / 0.004250 (0.060945) | 0.058951 / 0.037052 (0.021899) | 0.414466 / 0.258489 (0.155977) | 0.453844 / 0.293841 (0.160003) | 0.032437 / 0.128546 (-0.096110) | 0.008805 / 0.075646 (-0.066841) | 0.071741 / 0.419271 (-0.347530) | 0.048051 / 0.043533 (0.004518) | 0.413197 / 0.255139 (0.158058) | 0.430071 / 0.283200 (0.146872) | 0.023144 / 0.141683 (-0.118539) | 1.507756 / 1.452155 (0.055601) | 1.572180 / 1.492716 (0.079464) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.326556 / 0.018006 (0.308550) | 0.533664 / 0.000490 (0.533174) | 0.007400 / 0.000200 (0.007200) | 0.000119 / 0.000054 (0.000065) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033397 / 0.037411 (-0.004014) | 0.092486 / 0.014526 (0.077960) | 0.108454 / 0.176557 (-0.068103) | 0.163885 / 0.737135 (-0.573250) | 0.109682 / 0.296338 (-0.186657) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.429283 / 0.215209 (0.214074) | 4.285774 / 2.077655 (2.208119) | 2.245646 / 1.504120 (0.741526) | 2.088460 / 1.541195 (0.547265) | 2.217908 / 1.468490 (0.749418) | 0.500126 / 4.584777 (-4.084651) | 3.640253 / 3.745712 (-0.105459) | 3.435069 / 5.269862 (-1.834793) | 2.158015 / 4.565676 (-2.407662) | 0.059087 / 0.424275 (-0.365188) | 0.007479 / 0.007607 (-0.000128) | 0.518067 / 0.226044 (0.292023) | 5.181891 / 2.268929 (2.912963) | 2.759156 / 55.444624 (-52.685468) | 2.452164 / 6.876477 (-4.424313) | 2.712764 / 2.142072 (0.570692) | 0.604871 / 4.805227 (-4.200356) | 0.137810 / 6.500664 (-6.362854) | 0.061999 / 0.075469 (-0.013470) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.338081 / 1.841788 (-0.503706) | 19.934668 / 8.074308 (11.860360) | 14.482526 / 10.191392 (4.291134) | 0.167615 / 0.680424 (-0.512809) | 0.020257 / 0.534201 (-0.513944) | 0.399103 / 0.579283 (-0.180180) | 0.431785 / 0.434364 (-0.002579) | 0.475470 / 0.540337 (-0.064868) | 0.648003 / 1.386936 (-0.738933) |

</details>
</details>

![](https://cml.dev/watermark.png#153a6c0b0897fc30203ded5a6d6c358c53aa3a0e "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.011916 / 0.011353 (0.000563) | 0.004696 / 0.011008 (-0.006313) | 0.101061 / 0.038508 (0.062553) | 0.093383 / 0.023109 (0.070274) | 0.391517 / 0.275898 (0.115619) | 0.434374 / 0.323480 (0.110894) | 0.006193 / 0.007986 (-0.001792) | 0.003840 / 0.004328 (-0.000489) | 0.077946 / 0.004250 (0.073696) | 0.066332 / 0.037052 (0.029280) | 0.413103 / 0.258489 (0.154614) | 0.452988 / 0.293841 (0.159148) | 0.044899 / 0.128546 (-0.083647) | 0.009969 / 0.075646 (-0.065677) | 0.344569 / 0.419271 (-0.074703) | 0.064688 / 0.043533 (0.021155) | 0.388042 / 0.255139 (0.132903) | 0.417615 / 0.283200 (0.134416) | 0.032899 / 0.141683 (-0.108784) | 1.738834 / 1.452155 (0.286679) | 1.837562 / 1.492716 (0.344845) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.255265 / 0.018006 (0.237259) | 0.547550 / 0.000490 (0.547061) | 0.009018 / 0.000200 (0.008818) | 0.001232 / 0.000054 (0.001178) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033171 / 0.037411 (-0.004241) | 0.102569 / 0.014526 (0.088043) | 0.113611 / 0.176557 (-0.062946) | 0.181805 / 0.737135 (-0.555330) | 0.115015 / 0.296338 (-0.181323) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.456430 / 0.215209 (0.241221) | 4.536000 / 2.077655 (2.458346) | 2.220554 / 1.504120 (0.716434) | 2.037965 / 1.541195 (0.496770) | 2.223780 / 1.468490 (0.755290) | 0.565732 / 4.584777 (-4.019045) | 4.574917 / 3.745712 (0.829205) | 4.085683 / 5.269862 (-1.184178) | 2.529052 / 4.565676 (-2.036624) | 0.067061 / 0.424275 (-0.357214) | 0.009161 / 0.007607 (0.001554) | 0.551377 / 0.226044 (0.325332) | 5.510422 / 2.268929 (3.241493) | 2.788264 / 55.444624 (-52.656360) | 2.432821 / 6.876477 (-4.443656) | 2.500835 / 2.142072 (0.358762) | 0.683645 / 4.805227 (-4.121582) | 0.155595 / 6.500664 (-6.345069) | 0.072265 / 0.075469 (-0.003204) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.512571 / 1.841788 (-0.329217) | 23.752582 / 8.074308 (15.678273) | 16.798834 / 10.191392 (6.607442) | 0.210325 / 0.680424 (-0.470099) | 0.023446 / 0.534201 (-0.510755) | 0.472964 / 0.579283 (-0.106319) | 0.518003 / 0.434364 (0.083639) | 0.588422 / 0.540337 (0.048085) | 0.830762 / 1.386936 (-0.556174) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008075 / 0.011353 (-0.003278) | 0.004569 / 0.011008 (-0.006439) | 0.079786 / 0.038508 (0.041278) | 0.092741 / 0.023109 (0.069632) | 0.500732 / 0.275898 (0.224834) | 0.544108 / 0.323480 (0.220628) | 0.006305 / 0.007986 (-0.001680) | 0.003843 / 0.004328 (-0.000486) | 0.078347 / 0.004250 (0.074096) | 0.066969 / 0.037052 (0.029916) | 0.504116 / 0.258489 (0.245627) | 0.548109 / 0.293841 (0.254268) | 0.038263 / 0.128546 (-0.090283) | 0.010006 / 0.075646 (-0.065640) | 0.085582 / 0.419271 (-0.333690) | 0.056937 / 0.043533 (0.013404) | 0.502861 / 0.255139 (0.247722) | 0.532002 / 0.283200 (0.248802) | 0.027003 / 0.141683 (-0.114679) | 1.811658 / 1.452155 (0.359503) | 1.878863 / 1.492716 (0.386147) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.242297 / 0.018006 (0.224291) | 0.489060 / 0.000490 (0.488570) | 0.005770 / 0.000200 (0.005570) | 0.000129 / 0.000054 (0.000075) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.040368 / 0.037411 (0.002956) | 0.116221 / 0.014526 (0.101695) | 0.125195 / 0.176557 (-0.051361) | 0.188616 / 0.737135 (-0.548519) | 0.126473 / 0.296338 (-0.169866) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.513975 / 0.215209 (0.298766) | 5.122407 / 2.077655 (3.044752) | 2.854024 / 1.504120 (1.349904) | 2.611101 / 1.541195 (1.069906) | 2.704880 / 1.468490 (1.236390) | 0.581568 / 4.584777 (-4.003209) | 4.628965 / 3.745712 (0.883253) | 4.069359 / 5.269862 (-1.200503) | 2.433793 / 4.565676 (-2.131883) | 0.068624 / 0.424275 (-0.355651) | 0.008843 / 0.007607 (0.001235) | 0.609147 / 0.226044 (0.383102) | 6.096923 / 2.268929 (3.827995) | 3.411687 / 55.444624 (-52.032937) | 2.972037 / 6.876477 (-3.904440) | 3.210266 / 2.142072 (1.068194) | 0.697935 / 4.805227 (-4.107292) | 0.156855 / 6.500664 (-6.343809) | 0.072600 / 0.075469 (-0.002869) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.673126 / 1.841788 (-0.168661) | 24.782231 / 8.074308 (16.707923) | 17.945937 / 10.191392 (7.754545) | 0.229063 / 0.680424 (-0.451361) | 0.024264 / 0.534201 (-0.509937) | 0.474904 / 0.579283 (-0.104379) | 0.616602 / 0.434364 (0.182238) | 0.587687 / 0.540337 (0.047350) | 0.875600 / 1.386936 (-0.511336) |

</details>
</details>

![](https://cml.dev/watermark.png#a46ae63ad72c0733c947fa0f2996fa739d80e1ef "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004866 / 0.011353 (-0.006487) | 0.002877 / 0.011008 (-0.008132) | 0.061786 / 0.038508 (0.023277) | 0.051555 / 0.023109 (0.028446) | 0.262182 / 0.275898 (-0.013716) | 0.288908 / 0.323480 (-0.034572) | 0.002929 / 0.007986 (-0.005057) | 0.002358 / 0.004328 (-0.001971) | 0.048246 / 0.004250 (0.043995) | 0.040391 / 0.037052 (0.003339) | 0.268165 / 0.258489 (0.009675) | 0.304844 / 0.293841 (0.011003) | 0.023280 / 0.128546 (-0.105266) | 0.007274 / 0.075646 (-0.068372) | 0.200698 / 0.419271 (-0.218574) | 0.036181 / 0.043533 (-0.007352) | 0.267292 / 0.255139 (0.012153) | 0.286981 / 0.283200 (0.003781) | 0.018686 / 0.141683 (-0.122996) | 1.131903 / 1.452155 (-0.320251) | 1.196631 / 1.492716 (-0.296086) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092158 / 0.018006 (0.074152) | 0.300621 / 0.000490 (0.300132) | 0.000205 / 0.000200 (0.000006) | 0.000041 / 0.000054 (-0.000013) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018101 / 0.037411 (-0.019310) | 0.062478 / 0.014526 (0.047952) | 0.073092 / 0.176557 (-0.103464) | 0.119397 / 0.737135 (-0.617738) | 0.073768 / 0.296338 (-0.222570) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.286711 / 0.215209 (0.071502) | 2.766663 / 2.077655 (0.689008) | 1.431238 / 1.504120 (-0.072882) | 1.308312 / 1.541195 (-0.232883) | 1.344886 / 1.468490 (-0.123605) | 0.396719 / 4.584777 (-4.188058) | 2.371154 / 3.745712 (-1.374558) | 2.626471 / 5.269862 (-2.643391) | 1.574837 / 4.565676 (-2.990840) | 0.046344 / 0.424275 (-0.377931) | 0.005108 / 0.007607 (-0.002499) | 0.334200 / 0.226044 (0.108156) | 3.277034 / 2.268929 (1.008106) | 1.789338 / 55.444624 (-53.655286) | 1.527584 / 6.876477 (-5.348892) | 1.570417 / 2.142072 (-0.571656) | 0.472663 / 4.805227 (-4.332564) | 0.100825 / 6.500664 (-6.399839) | 0.042270 / 0.075469 (-0.033199) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.965416 / 1.841788 (-0.876372) | 11.827406 / 8.074308 (3.753098) | 10.820703 / 10.191392 (0.629311) | 0.128636 / 0.680424 (-0.551788) | 0.014696 / 0.534201 (-0.519505) | 0.271019 / 0.579283 (-0.308264) | 0.270077 / 0.434364 (-0.164287) | 0.313054 / 0.540337 (-0.227284) | 0.402941 / 1.386936 (-0.983995) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005204 / 0.011353 (-0.006149) | 0.002976 / 0.011008 (-0.008032) | 0.047723 / 0.038508 (0.009215) | 0.056180 / 0.023109 (0.033071) | 0.277751 / 0.275898 (0.001853) | 0.304109 / 0.323480 (-0.019371) | 0.004254 / 0.007986 (-0.003732) | 0.002386 / 0.004328 (-0.001943) | 0.047815 / 0.004250 (0.043564) | 0.041553 / 0.037052 (0.004501) | 0.280958 / 0.258489 (0.022469) | 0.308639 / 0.293841 (0.014799) | 0.023549 / 0.128546 (-0.104997) | 0.007846 / 0.075646 (-0.067800) | 0.053762 / 0.419271 (-0.365509) | 0.031763 / 0.043533 (-0.011770) | 0.278208 / 0.255139 (0.023069) | 0.294024 / 0.283200 (0.010825) | 0.018648 / 0.141683 (-0.123035) | 1.140664 / 1.452155 (-0.311490) | 1.206706 / 1.492716 (-0.286010) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093211 / 0.018006 (0.075205) | 0.303067 / 0.000490 (0.302577) | 0.000222 / 0.000200 (0.000022) | 0.000055 / 0.000054 (0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021745 / 0.037411 (-0.015666) | 0.070400 / 0.014526 (0.055874) | 0.083250 / 0.176557 (-0.093307) | 0.119745 / 0.737135 (-0.617391) | 0.083004 / 0.296338 (-0.213335) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.305841 / 0.215209 (0.090632) | 2.958171 / 2.077655 (0.880516) | 1.596990 / 1.504120 (0.092870) | 1.466522 / 1.541195 (-0.074673) | 1.487050 / 1.468490 (0.018560) | 0.402866 / 4.584777 (-4.181911) | 2.425415 / 3.745712 (-1.320297) | 2.545245 / 5.269862 (-2.724617) | 1.569719 / 4.565676 (-2.995958) | 0.046344 / 0.424275 (-0.377931) | 0.005275 / 0.007607 (-0.002332) | 0.362024 / 0.226044 (0.135980) | 3.556721 / 2.268929 (1.287792) | 1.961359 / 55.444624 (-53.483266) | 1.672835 / 6.876477 (-5.203641) | 1.814036 / 2.142072 (-0.328036) | 0.482012 / 4.805227 (-4.323215) | 0.099275 / 6.500664 (-6.401389) | 0.040988 / 0.075469 (-0.034481) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.984368 / 1.841788 (-0.857420) | 12.251555 / 8.074308 (4.177247) | 10.645975 / 10.191392 (0.454583) | 0.128955 / 0.680424 (-0.551468) | 0.015355 / 0.534201 (-0.518846) | 0.272498 / 0.579283 (-0.306785) | 0.279342 / 0.434364 (-0.155022) | 0.303055 / 0.540337 (-0.237282) | 0.392437 / 1.386936 (-0.994499) |

</details>
</details>

![](https://cml.dev/watermark.png#56458fa98d2670ff6bf47a782b6f418785c017fd "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009502 / 0.011353 (-0.001851) | 0.004957 / 0.011008 (-0.006052) | 0.111062 / 0.038508 (0.072553) | 0.100012 / 0.023109 (0.076903) | 0.415747 / 0.275898 (0.139849) | 0.453910 / 0.323480 (0.130430) | 0.006030 / 0.007986 (-0.001956) | 0.004271 / 0.004328 (-0.000057) | 0.088694 / 0.004250 (0.084444) | 0.064529 / 0.037052 (0.027477) | 0.414999 / 0.258489 (0.156510) | 0.477115 / 0.293841 (0.183274) | 0.047565 / 0.128546 (-0.080982) | 0.013352 / 0.075646 (-0.062294) | 0.367948 / 0.419271 (-0.051324) | 0.067577 / 0.043533 (0.024044) | 0.405107 / 0.255139 (0.149968) | 0.430281 / 0.283200 (0.147081) | 0.041629 / 0.141683 (-0.100054) | 1.784746 / 1.452155 (0.332591) | 1.901539 / 1.492716 (0.408822) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.308456 / 0.018006 (0.290450) | 0.623253 / 0.000490 (0.622763) | 0.014966 / 0.000200 (0.014766) | 0.000393 / 0.000054 (0.000338) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031538 / 0.037411 (-0.005873) | 0.100321 / 0.014526 (0.085796) | 0.112788 / 0.176557 (-0.063769) | 0.180998 / 0.737135 (-0.556138) | 0.111589 / 0.296338 (-0.184750) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.603121 / 0.215209 (0.387912) | 5.769795 / 2.077655 (3.692140) | 2.501168 / 1.504120 (0.997048) | 2.240982 / 1.541195 (0.699787) | 2.333123 / 1.468490 (0.864633) | 0.799246 / 4.584777 (-3.785531) | 5.148529 / 3.745712 (1.402817) | 4.737782 / 5.269862 (-0.532080) | 3.003032 / 4.565676 (-1.562644) | 0.087457 / 0.424275 (-0.336818) | 0.008777 / 0.007607 (0.001170) | 0.692961 / 0.226044 (0.466916) | 7.235537 / 2.268929 (4.966608) | 3.464074 / 55.444624 (-51.980551) | 2.817360 / 6.876477 (-4.059116) | 2.903121 / 2.142072 (0.761049) | 1.026150 / 4.805227 (-3.779077) | 0.231814 / 6.500664 (-6.268850) | 0.088358 / 0.075469 (0.012888) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.527889 / 1.841788 (-0.313898) | 24.374770 / 8.074308 (16.300462) | 21.720415 / 10.191392 (11.529023) | 0.209357 / 0.680424 (-0.471067) | 0.027587 / 0.534201 (-0.506614) | 0.479136 / 0.579283 (-0.100147) | 0.573005 / 0.434364 (0.138641) | 0.537713 / 0.540337 (-0.002625) | 0.753628 / 1.386936 (-0.633308) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009724 / 0.011353 (-0.001629) | 0.004798 / 0.011008 (-0.006210) | 0.076423 / 0.038508 (0.037915) | 0.085693 / 0.023109 (0.062584) | 0.446864 / 0.275898 (0.170966) | 0.482700 / 0.323480 (0.159220) | 0.006448 / 0.007986 (-0.001537) | 0.004451 / 0.004328 (0.000122) | 0.078295 / 0.004250 (0.074045) | 0.061940 / 0.037052 (0.024888) | 0.446091 / 0.258489 (0.187601) | 0.478567 / 0.293841 (0.184726) | 0.047206 / 0.128546 (-0.081340) | 0.012608 / 0.075646 (-0.063038) | 0.089719 / 0.419271 (-0.329552) | 0.057791 / 0.043533 (0.014258) | 0.438357 / 0.255139 (0.183218) | 0.475060 / 0.283200 (0.191860) | 0.035466 / 0.141683 (-0.106216) | 1.691982 / 1.452155 (0.239827) | 1.773834 / 1.492716 (0.281118) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.290053 / 0.018006 (0.272047) | 0.595465 / 0.000490 (0.594976) | 0.007531 / 0.000200 (0.007331) | 0.000179 / 0.000054 (0.000124) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034625 / 0.037411 (-0.002786) | 0.098725 / 0.014526 (0.084200) | 0.111248 / 0.176557 (-0.065308) | 0.172113 / 0.737135 (-0.565022) | 0.111299 / 0.296338 (-0.185040) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.581773 / 0.215209 (0.366564) | 6.150993 / 2.077655 (4.073338) | 2.761099 / 1.504120 (1.256980) | 2.431459 / 1.541195 (0.890264) | 2.501471 / 1.468490 (1.032981) | 0.805751 / 4.584777 (-3.779026) | 5.375406 / 3.745712 (1.629693) | 4.829323 / 5.269862 (-0.440538) | 3.095235 / 4.565676 (-1.470442) | 0.103336 / 0.424275 (-0.320939) | 0.012678 / 0.007607 (0.005071) | 0.730121 / 0.226044 (0.504077) | 7.272025 / 2.268929 (5.003097) | 3.607889 / 55.444624 (-51.836735) | 2.904797 / 6.876477 (-3.971680) | 3.179139 / 2.142072 (1.037067) | 0.997510 / 4.805227 (-3.807717) | 0.219023 / 6.500664 (-6.281641) | 0.076680 / 0.075469 (0.001211) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.712838 / 1.841788 (-0.128950) | 24.242240 / 8.074308 (16.167932) | 19.746825 / 10.191392 (9.555433) | 0.234590 / 0.680424 (-0.445833) | 0.032015 / 0.534201 (-0.502186) | 0.462554 / 0.579283 (-0.116729) | 0.604529 / 0.434364 (0.170165) | 0.537779 / 0.540337 (-0.002558) | 0.777386 / 1.386936 (-0.609550) |

</details>
</details>

![](https://cml.dev/watermark.png#89c1b13d85b8971925440deb84f558e23c224a47 "CML watermark")


Cool ! Nice to simplify this

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004659 / 0.011353 (-0.006693) | 0.002672 / 0.011008 (-0.008337) | 0.062385 / 0.038508 (0.023877) | 0.030581 / 0.023109 (0.007471) | 0.243210 / 0.275898 (-0.032688) | 0.271441 / 0.323480 (-0.052039) | 0.002909 / 0.007986 (-0.005076) | 0.002371 / 0.004328 (-0.001957) | 0.049213 / 0.004250 (0.044962) | 0.043952 / 0.037052 (0.006900) | 0.250257 / 0.258489 (-0.008232) | 0.280470 / 0.293841 (-0.013371) | 0.023048 / 0.128546 (-0.105499) | 0.006893 / 0.075646 (-0.068754) | 0.204026 / 0.419271 (-0.215245) | 0.054067 / 0.043533 (0.010534) | 0.248730 / 0.255139 (-0.006409) | 0.272325 / 0.283200 (-0.010874) | 0.019028 / 0.141683 (-0.122655) | 1.103477 / 1.452155 (-0.348678) | 1.185775 / 1.492716 (-0.306942) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.097295 / 0.018006 (0.079289) | 0.302997 / 0.000490 (0.302507) | 0.000216 / 0.000200 (0.000016) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018653 / 0.037411 (-0.018759) | 0.062604 / 0.014526 (0.048079) | 0.075652 / 0.176557 (-0.100904) | 0.121298 / 0.737135 (-0.615838) | 0.074129 / 0.296338 (-0.222209) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.283315 / 0.215209 (0.068106) | 2.833975 / 2.077655 (0.756320) | 1.463877 / 1.504120 (-0.040243) | 1.352197 / 1.541195 (-0.188998) | 1.337623 / 1.468490 (-0.130867) | 0.405282 / 4.584777 (-4.179495) | 2.371381 / 3.745712 (-1.374331) | 2.584853 / 5.269862 (-2.685009) | 1.565902 / 4.565676 (-2.999775) | 0.046398 / 0.424275 (-0.377877) | 0.004795 / 0.007607 (-0.002812) | 0.345949 / 0.226044 (0.119905) | 3.326662 / 2.268929 (1.057733) | 1.778394 / 55.444624 (-53.666230) | 1.520788 / 6.876477 (-5.355688) | 1.526517 / 2.142072 (-0.615556) | 0.471788 / 4.805227 (-4.333439) | 0.099236 / 6.500664 (-6.401428) | 0.041886 / 0.075469 (-0.033583) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.958183 / 1.841788 (-0.883605) | 11.474476 / 8.074308 (3.400168) | 10.547550 / 10.191392 (0.356158) | 0.129316 / 0.680424 (-0.551108) | 0.013969 / 0.534201 (-0.520232) | 0.272028 / 0.579283 (-0.307255) | 0.271027 / 0.434364 (-0.163337) | 0.312124 / 0.540337 (-0.228214) | 0.423879 / 1.386936 (-0.963057) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004743 / 0.011353 (-0.006610) | 0.002724 / 0.011008 (-0.008284) | 0.049526 / 0.038508 (0.011018) | 0.051429 / 0.023109 (0.028319) | 0.265202 / 0.275898 (-0.010696) | 0.287498 / 0.323480 (-0.035981) | 0.004034 / 0.007986 (-0.003951) | 0.002460 / 0.004328 (-0.001868) | 0.049367 / 0.004250 (0.045116) | 0.038526 / 0.037052 (0.001474) | 0.271496 / 0.258489 (0.013007) | 0.300969 / 0.293841 (0.007128) | 0.024159 / 0.128546 (-0.104387) | 0.006959 / 0.075646 (-0.068687) | 0.055316 / 0.419271 (-0.363955) | 0.032409 / 0.043533 (-0.011124) | 0.267524 / 0.255139 (0.012385) | 0.284667 / 0.283200 (0.001467) | 0.017305 / 0.141683 (-0.124378) | 1.127560 / 1.452155 (-0.324595) | 1.188271 / 1.492716 (-0.304445) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093587 / 0.018006 (0.075581) | 0.301834 / 0.000490 (0.301344) | 0.000211 / 0.000200 (0.000011) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.020899 / 0.037411 (-0.016512) | 0.069999 / 0.014526 (0.055473) | 0.081434 / 0.176557 (-0.095123) | 0.120538 / 0.737135 (-0.616598) | 0.082708 / 0.296338 (-0.213630) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.291845 / 0.215209 (0.076636) | 2.872476 / 2.077655 (0.794822) | 1.579330 / 1.504120 (0.075210) | 1.453083 / 1.541195 (-0.088112) | 1.496675 / 1.468490 (0.028185) | 0.406178 / 4.584777 (-4.178599) | 2.434121 / 3.745712 (-1.311592) | 2.519760 / 5.269862 (-2.750101) | 1.535781 / 4.565676 (-3.029895) | 0.046331 / 0.424275 (-0.377944) | 0.004749 / 0.007607 (-0.002858) | 0.340862 / 0.226044 (0.114817) | 3.362750 / 2.268929 (1.093822) | 1.924707 / 55.444624 (-53.519917) | 1.646820 / 6.876477 (-5.229657) | 1.630885 / 2.142072 (-0.511188) | 0.478623 / 4.805227 (-4.326605) | 0.098235 / 6.500664 (-6.402429) | 0.040741 / 0.075469 (-0.034728) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.989858 / 1.841788 (-0.851929) | 12.111035 / 8.074308 (4.036727) | 11.065284 / 10.191392 (0.873892) | 0.143443 / 0.680424 (-0.536981) | 0.015873 / 0.534201 (-0.518328) | 0.271932 / 0.579283 (-0.307351) | 0.281440 / 0.434364 (-0.152924) | 0.309518 / 0.540337 (-0.230819) | 0.414701 / 1.386936 (-0.972235) |

</details>
</details>

![](https://cml.dev/watermark.png#322ee4bd7d460a5789f9991a45453b9fb5f5aed1 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005840 / 0.011353 (-0.005513) | 0.003580 / 0.011008 (-0.007428) | 0.079921 / 0.038508 (0.041413) | 0.036316 / 0.023109 (0.013206) | 0.321065 / 0.275898 (0.045167) | 0.348594 / 0.323480 (0.025115) | 0.004662 / 0.007986 (-0.003324) | 0.002884 / 0.004328 (-0.001444) | 0.062964 / 0.004250 (0.058714) | 0.052856 / 0.037052 (0.015804) | 0.322087 / 0.258489 (0.063598) | 0.355546 / 0.293841 (0.061705) | 0.027025 / 0.128546 (-0.101521) | 0.007969 / 0.075646 (-0.067678) | 0.261416 / 0.419271 (-0.157855) | 0.066612 / 0.043533 (0.023079) | 0.314631 / 0.255139 (0.059492) | 0.340939 / 0.283200 (0.057739) | 0.019710 / 0.141683 (-0.121972) | 1.446068 / 1.452155 (-0.006086) | 1.510342 / 1.492716 (0.017625) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.219742 / 0.018006 (0.201736) | 0.431794 / 0.000490 (0.431304) | 0.005717 / 0.000200 (0.005517) | 0.000195 / 0.000054 (0.000141) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024486 / 0.037411 (-0.012926) | 0.073231 / 0.014526 (0.058706) | 0.084053 / 0.176557 (-0.092503) | 0.145857 / 0.737135 (-0.591279) | 0.083050 / 0.296338 (-0.213289) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.400532 / 0.215209 (0.185323) | 3.989293 / 2.077655 (1.911638) | 1.935520 / 1.504120 (0.431400) | 1.754146 / 1.541195 (0.212951) | 1.821060 / 1.468490 (0.352570) | 0.512603 / 4.584777 (-4.072173) | 3.070974 / 3.745712 (-0.674738) | 2.984617 / 5.269862 (-2.285245) | 1.875790 / 4.565676 (-2.689886) | 0.057881 / 0.424275 (-0.366394) | 0.006403 / 0.007607 (-0.001204) | 0.465542 / 0.226044 (0.239498) | 4.659589 / 2.268929 (2.390661) | 2.349637 / 55.444624 (-53.094987) | 2.011511 / 6.876477 (-4.864965) | 2.071893 / 2.142072 (-0.070179) | 0.591113 / 4.805227 (-4.214114) | 0.125000 / 6.500664 (-6.375664) | 0.061372 / 0.075469 (-0.014097) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.237068 / 1.841788 (-0.604720) | 17.493192 / 8.074308 (9.418884) | 13.600688 / 10.191392 (3.409296) | 0.142508 / 0.680424 (-0.537916) | 0.017305 / 0.534201 (-0.516896) | 0.333352 / 0.579283 (-0.245931) | 0.366699 / 0.434364 (-0.067665) | 0.381104 / 0.540337 (-0.159233) | 0.562645 / 1.386936 (-0.824291) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006337 / 0.011353 (-0.005016) | 0.003584 / 0.011008 (-0.007424) | 0.063351 / 0.038508 (0.024843) | 0.061351 / 0.023109 (0.038242) | 0.430690 / 0.275898 (0.154792) | 0.462158 / 0.323480 (0.138678) | 0.004922 / 0.007986 (-0.003064) | 0.002898 / 0.004328 (-0.001430) | 0.063722 / 0.004250 (0.059472) | 0.046970 / 0.037052 (0.009918) | 0.436340 / 0.258489 (0.177851) | 0.472842 / 0.293841 (0.179001) | 0.029238 / 0.128546 (-0.099309) | 0.008079 / 0.075646 (-0.067568) | 0.068425 / 0.419271 (-0.350846) | 0.041272 / 0.043533 (-0.002261) | 0.429150 / 0.255139 (0.174011) | 0.451859 / 0.283200 (0.168659) | 0.020135 / 0.141683 (-0.121547) | 1.440388 / 1.452155 (-0.011767) | 1.506784 / 1.492716 (0.014068) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.225810 / 0.018006 (0.207804) | 0.408447 / 0.000490 (0.407957) | 0.002484 / 0.000200 (0.002284) | 0.000079 / 0.000054 (0.000024) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026162 / 0.037411 (-0.011250) | 0.079292 / 0.014526 (0.064766) | 0.091126 / 0.176557 (-0.085431) | 0.141607 / 0.737135 (-0.595528) | 0.090073 / 0.296338 (-0.206266) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.420689 / 0.215209 (0.205479) | 4.207631 / 2.077655 (2.129976) | 2.163469 / 1.504120 (0.659350) | 2.098208 / 1.541195 (0.557013) | 2.217340 / 1.468490 (0.748850) | 0.502599 / 4.584777 (-4.082178) | 3.128151 / 3.745712 (-0.617561) | 2.921041 / 5.269862 (-2.348820) | 1.808352 / 4.565676 (-2.757325) | 0.057724 / 0.424275 (-0.366551) | 0.006423 / 0.007607 (-0.001184) | 0.490631 / 0.226044 (0.264587) | 4.878761 / 2.268929 (2.609833) | 2.614831 / 55.444624 (-52.829793) | 2.214611 / 6.876477 (-4.661866) | 2.253313 / 2.142072 (0.111241) | 0.585643 / 4.805227 (-4.219584) | 0.122436 / 6.500664 (-6.378228) | 0.057974 / 0.075469 (-0.017495) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.334290 / 1.841788 (-0.507498) | 17.778981 / 8.074308 (9.704672) | 14.982837 / 10.191392 (4.791445) | 0.135731 / 0.680424 (-0.544693) | 0.018314 / 0.534201 (-0.515887) | 0.332318 / 0.579283 (-0.246966) | 0.380185 / 0.434364 (-0.054179) | 0.391430 / 0.540337 (-0.148907) | 0.554577 / 1.386936 (-0.832359) |

</details>
</details>

![](https://cml.dev/watermark.png#1715b61a096cafb20caeb136111522432aba04f5 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005248 / 0.011353 (-0.006105) | 0.003188 / 0.011008 (-0.007820) | 0.063045 / 0.038508 (0.024537) | 0.033620 / 0.023109 (0.010511) | 0.244725 / 0.275898 (-0.031173) | 0.283259 / 0.323480 (-0.040220) | 0.003013 / 0.007986 (-0.004973) | 0.002486 / 0.004328 (-0.001842) | 0.048873 / 0.004250 (0.044623) | 0.049431 / 0.037052 (0.012379) | 0.245297 / 0.258489 (-0.013192) | 0.283127 / 0.293841 (-0.010714) | 0.024204 / 0.128546 (-0.104342) | 0.007542 / 0.075646 (-0.068104) | 0.204831 / 0.419271 (-0.214440) | 0.067487 / 0.043533 (0.023954) | 0.251477 / 0.255139 (-0.003662) | 0.273108 / 0.283200 (-0.010091) | 0.021035 / 0.141683 (-0.120648) | 1.108361 / 1.452155 (-0.343793) | 1.172923 / 1.492716 (-0.319793) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094729 / 0.018006 (0.076722) | 0.301877 / 0.000490 (0.301388) | 0.000223 / 0.000200 (0.000023) | 0.000050 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019901 / 0.037411 (-0.017511) | 0.068059 / 0.014526 (0.053534) | 0.075333 / 0.176557 (-0.101224) | 0.123276 / 0.737135 (-0.613859) | 0.076810 / 0.296338 (-0.219528) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.283421 / 0.215209 (0.068211) | 2.775511 / 2.077655 (0.697857) | 1.430927 / 1.504120 (-0.073193) | 1.317334 / 1.541195 (-0.223860) | 1.359483 / 1.468490 (-0.109007) | 0.403186 / 4.584777 (-4.181591) | 2.405789 / 3.745712 (-1.339923) | 2.773039 / 5.269862 (-2.496823) | 1.666722 / 4.565676 (-2.898954) | 0.047937 / 0.424275 (-0.376338) | 0.004879 / 0.007607 (-0.002728) | 0.347225 / 0.226044 (0.121180) | 3.380860 / 2.268929 (1.111931) | 1.838532 / 55.444624 (-53.606092) | 1.597681 / 6.876477 (-5.278796) | 1.600123 / 2.142072 (-0.541949) | 0.478836 / 4.805227 (-4.326391) | 0.100332 / 6.500664 (-6.400332) | 0.043334 / 0.075469 (-0.032135) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.942591 / 1.841788 (-0.899196) | 12.588886 / 8.074308 (4.514578) | 11.375666 / 10.191392 (1.184274) | 0.143460 / 0.680424 (-0.536964) | 0.014990 / 0.534201 (-0.519211) | 0.271068 / 0.579283 (-0.308216) | 0.265478 / 0.434364 (-0.168885) | 0.310914 / 0.540337 (-0.229423) | 0.428310 / 1.386936 (-0.958626) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004986 / 0.011353 (-0.006367) | 0.003263 / 0.011008 (-0.007745) | 0.049076 / 0.038508 (0.010567) | 0.063665 / 0.023109 (0.040556) | 0.270352 / 0.275898 (-0.005546) | 0.298849 / 0.323480 (-0.024631) | 0.004083 / 0.007986 (-0.003903) | 0.002503 / 0.004328 (-0.001826) | 0.048586 / 0.004250 (0.044335) | 0.040701 / 0.037052 (0.003648) | 0.274082 / 0.258489 (0.015593) | 0.308279 / 0.293841 (0.014438) | 0.024734 / 0.128546 (-0.103812) | 0.007535 / 0.075646 (-0.068111) | 0.054670 / 0.419271 (-0.364602) | 0.032828 / 0.043533 (-0.010705) | 0.276226 / 0.255139 (0.021087) | 0.289322 / 0.283200 (0.006122) | 0.018789 / 0.141683 (-0.122893) | 1.279837 / 1.452155 (-0.172318) | 1.203010 / 1.492716 (-0.289706) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095674 / 0.018006 (0.077667) | 0.309754 / 0.000490 (0.309265) | 0.000229 / 0.000200 (0.000029) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021733 / 0.037411 (-0.015678) | 0.074858 / 0.014526 (0.060332) | 0.081845 / 0.176557 (-0.094711) | 0.121991 / 0.737135 (-0.615145) | 0.084057 / 0.296338 (-0.212281) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.298456 / 0.215209 (0.083246) | 2.884930 / 2.077655 (0.807276) | 1.574875 / 1.504120 (0.070755) | 1.451598 / 1.541195 (-0.089597) | 1.548106 / 1.468490 (0.079616) | 0.408662 / 4.584777 (-4.176115) | 2.444306 / 3.745712 (-1.301406) | 2.737027 / 5.269862 (-2.532835) | 1.633085 / 4.565676 (-2.932592) | 0.047349 / 0.424275 (-0.376926) | 0.004864 / 0.007607 (-0.002744) | 0.355434 / 0.226044 (0.129389) | 3.495531 / 2.268929 (1.226603) | 1.972737 / 55.444624 (-53.471888) | 1.706973 / 6.876477 (-5.169504) | 1.798985 / 2.142072 (-0.343087) | 0.490353 / 4.805227 (-4.314874) | 0.099533 / 6.500664 (-6.401131) | 0.042397 / 0.075469 (-0.033073) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.978092 / 1.841788 (-0.863696) | 13.166220 / 8.074308 (5.091912) | 11.673518 / 10.191392 (1.482126) | 0.134253 / 0.680424 (-0.546171) | 0.016478 / 0.534201 (-0.517723) | 0.271629 / 0.579283 (-0.307654) | 0.284082 / 0.434364 (-0.150282) | 0.313352 / 0.540337 (-0.226986) | 0.416913 / 1.386936 (-0.970023) |

</details>
</details>

![](https://cml.dev/watermark.png#01e7144a02825fea3418872c51a8ca93950f3080 "CML watermark")