Issue: Ensure last tqdm update in `map`
State: closed
Author: mariosasko

This PR modifies `map` to:
* ensure the TQDM bar gets the last progress update
* when a map function fails, avoid throwing a chained exception in the single-proc mode

Comments:
<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.011060 / 0.011353 (-0.000293) | 0.005752 / 0.011008 (-0.005256) | 0.120349 / 0.038508 (0.081841) | 0.045303 / 0.023109 (0.022194) | 0.359196 / 0.275898 (0.083298) | 0.406351 / 0.323480 (0.082871) | 0.009474 / 0.007986 (0.001489) | 0.004524 / 0.004328 (0.000195) | 0.091990 / 0.004250 (0.087739) | 0.050034 / 0.037052 (0.012982) | 0.372479 / 0.258489 (0.113990) | 0.418907 / 0.293841 (0.125067) | 0.044300 / 0.128546 (-0.084247) | 0.013989 / 0.075646 (-0.061657) | 0.397406 / 0.419271 (-0.021866) | 0.056070 / 0.043533 (0.012537) | 0.357597 / 0.255139 (0.102458) | 0.382938 / 0.283200 (0.099738) | 0.117060 / 0.141683 (-0.024623) | 1.670869 / 1.452155 (0.218714) | 1.780944 / 1.492716 (0.288227) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.229578 / 0.018006 (0.211572) | 0.493711 / 0.000490 (0.493222) | 0.008413 / 0.000200 (0.008213) | 0.000118 / 0.000054 (0.000063) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033364 / 0.037411 (-0.004047) | 0.135953 / 0.014526 (0.121427) | 0.141942 / 0.176557 (-0.034614) | 0.225891 / 0.737135 (-0.511244) | 0.151010 / 0.296338 (-0.145328) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.470937 / 0.215209 (0.255728) | 4.710258 / 2.077655 (2.632603) | 2.132025 / 1.504120 (0.627905) | 1.913134 / 1.541195 (0.371939) | 2.025993 / 1.468490 (0.557503) | 0.835993 / 4.584777 (-3.748784) | 4.446678 / 3.745712 (0.700965) | 4.260014 / 5.269862 (-1.009847) | 2.193078 / 4.565676 (-2.372598) | 0.100132 / 0.424275 (-0.324143) | 0.014163 / 0.007607 (0.006556) | 0.599252 / 0.226044 (0.373208) | 5.976377 / 2.268929 (3.707448) | 2.678116 / 55.444624 (-52.766508) | 2.309311 / 6.876477 (-4.567166) | 2.410284 / 2.142072 (0.268212) | 1.002415 / 4.805227 (-3.802813) | 0.194588 / 6.500664 (-6.306076) | 0.074921 / 0.075469 (-0.000548) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.432389 / 1.841788 (-0.409399) | 17.915288 / 8.074308 (9.840980) | 17.190906 / 10.191392 (6.999514) | 0.238469 / 0.680424 (-0.441955) | 0.036270 / 0.534201 (-0.497931) | 0.537320 / 0.579283 (-0.041963) | 0.512876 / 0.434364 (0.078512) | 0.629022 / 0.540337 (0.088685) | 0.750109 / 1.386936 (-0.636827) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008544 / 0.011353 (-0.002809) | 0.005933 / 0.011008 (-0.005075) | 0.088879 / 0.038508 (0.050371) | 0.040387 / 0.023109 (0.017278) | 0.406392 / 0.275898 (0.130494) | 0.449572 / 0.323480 (0.126092) | 0.006623 / 0.007986 (-0.001362) | 0.004727 / 0.004328 (0.000398) | 0.086745 / 0.004250 (0.082495) | 0.054335 / 0.037052 (0.017283) | 0.405652 / 0.258489 (0.147163) | 0.473934 / 0.293841 (0.180093) | 0.042157 / 0.128546 (-0.086390) | 0.014249 / 0.075646 (-0.061397) | 0.102130 / 0.419271 (-0.317141) | 0.056815 / 0.043533 (0.013282) | 0.407945 / 0.255139 (0.152806) | 0.431720 / 0.283200 (0.148521) | 0.119901 / 0.141683 (-0.021781) | 1.738381 / 1.452155 (0.286227) | 1.838981 / 1.492716 (0.346265) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.251926 / 0.018006 (0.233919) | 0.498117 / 0.000490 (0.497627) | 0.000439 / 0.000200 (0.000239) | 0.000065 / 0.000054 (0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034526 / 0.037411 (-0.002886) | 0.133038 / 0.014526 (0.118512) | 0.147494 / 0.176557 (-0.029063) | 0.234392 / 0.737135 (-0.502743) | 0.152361 / 0.296338 (-0.143978) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.495144 / 0.215209 (0.279935) | 4.936646 / 2.077655 (2.858991) | 2.385549 / 1.504120 (0.881429) | 2.173817 / 1.541195 (0.632622) | 2.327508 / 1.468490 (0.859018) | 0.851899 / 4.584777 (-3.732878) | 4.820388 / 3.745712 (1.074676) | 2.500304 / 5.269862 (-2.769558) | 1.621246 / 4.565676 (-2.944430) | 0.102858 / 0.424275 (-0.321417) | 0.014719 / 0.007607 (0.007112) | 0.611880 / 0.226044 (0.385836) | 6.100737 / 2.268929 (3.831808) | 2.955681 / 55.444624 (-52.488943) | 2.563533 / 6.876477 (-4.312943) | 2.659030 / 2.142072 (0.516958) | 1.004737 / 4.805227 (-3.800490) | 0.198379 / 6.500664 (-6.302285) | 0.078705 / 0.075469 (0.003236) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.501155 / 1.841788 (-0.340633) | 18.381513 / 8.074308 (10.307205) | 16.173893 / 10.191392 (5.982501) | 0.209497 / 0.680424 (-0.470927) | 0.021640 / 0.534201 (-0.512561) | 0.505905 / 0.579283 (-0.073378) | 0.513446 / 0.434364 (0.079082) | 0.652704 / 0.540337 (0.112366) | 0.761038 / 1.386936 (-0.625898) |

</details>
</details>

![](https://cml.dev/watermark.png#b8235c92b46b6a63286fcee1a56adae4c0a751d3 "CML watermark")


_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009085 / 0.011353 (-0.002268) | 0.004589 / 0.011008 (-0.006419) | 0.100820 / 0.038508 (0.062312) | 0.030677 / 0.023109 (0.007568) | 0.306702 / 0.275898 (0.030804) | 0.360623 / 0.323480 (0.037144) | 0.007377 / 0.007986 (-0.000608) | 0.003480 / 0.004328 (-0.000848) | 0.077813 / 0.004250 (0.073562) | 0.037293 / 0.037052 (0.000241) | 0.314137 / 0.258489 (0.055648) | 0.343394 / 0.293841 (0.049554) | 0.034202 / 0.128546 (-0.094344) | 0.011417 / 0.075646 (-0.064230) | 0.322584 / 0.419271 (-0.096687) | 0.041524 / 0.043533 (-0.002009) | 0.308116 / 0.255139 (0.052977) | 0.324527 / 0.283200 (0.041327) | 0.090973 / 0.141683 (-0.050710) | 1.515941 / 1.452155 (0.063787) | 1.548975 / 1.492716 (0.056259) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.185901 / 0.018006 (0.167895) | 0.420742 / 0.000490 (0.420252) | 0.002958 / 0.000200 (0.002758) | 0.000086 / 0.000054 (0.000032) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024242 / 0.037411 (-0.013170) | 0.098827 / 0.014526 (0.084302) | 0.107609 / 0.176557 (-0.068947) | 0.172228 / 0.737135 (-0.564908) | 0.110042 / 0.296338 (-0.186296) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.429647 / 0.215209 (0.214438) | 4.265406 / 2.077655 (2.187751) | 1.924514 / 1.504120 (0.420394) | 1.709881 / 1.541195 (0.168686) | 1.764872 / 1.468490 (0.296382) | 0.698089 / 4.584777 (-3.886688) | 3.439154 / 3.745712 (-0.306558) | 1.925058 / 5.269862 (-3.344804) | 1.267506 / 4.565676 (-3.298171) | 0.082167 / 0.424275 (-0.342108) | 0.012450 / 0.007607 (0.004843) | 0.523077 / 0.226044 (0.297033) | 5.240422 / 2.268929 (2.971494) | 2.363666 / 55.444624 (-53.080959) | 2.021903 / 6.876477 (-4.854574) | 2.136430 / 2.142072 (-0.005643) | 0.816377 / 4.805227 (-3.988850) | 0.151516 / 6.500664 (-6.349148) | 0.066590 / 0.075469 (-0.008879) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.216477 / 1.841788 (-0.625310) | 13.685044 / 8.074308 (5.610736) | 14.082620 / 10.191392 (3.891228) | 0.148399 / 0.680424 (-0.532025) | 0.028337 / 0.534201 (-0.505864) | 0.405379 / 0.579283 (-0.173904) | 0.405650 / 0.434364 (-0.028714) | 0.492658 / 0.540337 (-0.047679) | 0.578836 / 1.386936 (-0.808100) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006863 / 0.011353 (-0.004490) | 0.004746 / 0.011008 (-0.006262) | 0.075802 / 0.038508 (0.037294) | 0.027950 / 0.023109 (0.004840) | 0.347613 / 0.275898 (0.071715) | 0.401201 / 0.323480 (0.077721) | 0.005765 / 0.007986 (-0.002221) | 0.003567 / 0.004328 (-0.000762) | 0.074188 / 0.004250 (0.069937) | 0.041209 / 0.037052 (0.004157) | 0.346541 / 0.258489 (0.088052) | 0.425729 / 0.293841 (0.131888) | 0.032430 / 0.128546 (-0.096116) | 0.011708 / 0.075646 (-0.063938) | 0.084667 / 0.419271 (-0.334604) | 0.042155 / 0.043533 (-0.001378) | 0.341210 / 0.255139 (0.086071) | 0.389759 / 0.283200 (0.106559) | 0.092640 / 0.141683 (-0.049042) | 1.526093 / 1.452155 (0.073938) | 1.556277 / 1.492716 (0.063561) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.232383 / 0.018006 (0.214377) | 0.412353 / 0.000490 (0.411863) | 0.004009 / 0.000200 (0.003809) | 0.000071 / 0.000054 (0.000017) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025854 / 0.037411 (-0.011557) | 0.102660 / 0.014526 (0.088134) | 0.108420 / 0.176557 (-0.068137) | 0.175834 / 0.737135 (-0.561301) | 0.113472 / 0.296338 (-0.182867) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.443595 / 0.215209 (0.228386) | 4.420959 / 2.077655 (2.343305) | 2.112790 / 1.504120 (0.608670) | 1.908836 / 1.541195 (0.367641) | 1.998340 / 1.468490 (0.529850) | 0.706096 / 4.584777 (-3.878681) | 3.400871 / 3.745712 (-0.344841) | 2.803315 / 5.269862 (-2.466547) | 1.539392 / 4.565676 (-3.026284) | 0.083523 / 0.424275 (-0.340752) | 0.012541 / 0.007607 (0.004934) | 0.543428 / 0.226044 (0.317383) | 5.467416 / 2.268929 (3.198488) | 2.551970 / 55.444624 (-52.892654) | 2.212708 / 6.876477 (-4.663768) | 2.266169 / 2.142072 (0.124096) | 0.809943 / 4.805227 (-3.995284) | 0.152300 / 6.500664 (-6.348364) | 0.068591 / 0.075469 (-0.006878) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.330141 / 1.841788 (-0.511646) | 14.292734 / 8.074308 (6.218426) | 13.556157 / 10.191392 (3.364765) | 0.155949 / 0.680424 (-0.524475) | 0.016464 / 0.534201 (-0.517737) | 0.377906 / 0.579283 (-0.201377) | 0.390385 / 0.434364 (-0.043979) | 0.471867 / 0.540337 (-0.068471) | 0.557794 / 1.386936 (-0.829142) |

</details>
</details>

![](https://cml.dev/watermark.png#ba50512b76ef315f73bf821b0487296cdb373850 "CML watermark")


I just tried on colab and it didn't finish the progress bar for some reason.

Maybe we need to call `pbar.close()` before `return`

<img width="729" alt="image" src="https://user-images.githubusercontent.com/42851186/220417517-919438a4-5462-4e87-8f84-e9399a9be27c.png">


(just added .close() - let me try quickly if it works now)

it worked ! :)

<img width="575" alt="image" src="https://user-images.githubusercontent.com/42851186/220419220-8108f225-13cb-4968-acff-fe4543d5a324.png">


<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008465 / 0.011353 (-0.002888) | 0.004622 / 0.011008 (-0.006387) | 0.100365 / 0.038508 (0.061857) | 0.029453 / 0.023109 (0.006344) | 0.358041 / 0.275898 (0.082143) | 0.424777 / 0.323480 (0.101298) | 0.006930 / 0.007986 (-0.001055) | 0.004756 / 0.004328 (0.000428) | 0.077128 / 0.004250 (0.072878) | 0.036338 / 0.037052 (-0.000715) | 0.367613 / 0.258489 (0.109124) | 0.397798 / 0.293841 (0.103957) | 0.033500 / 0.128546 (-0.095047) | 0.011427 / 0.075646 (-0.064219) | 0.321617 / 0.419271 (-0.097654) | 0.040937 / 0.043533 (-0.002596) | 0.345358 / 0.255139 (0.090219) | 0.366932 / 0.283200 (0.083733) | 0.086506 / 0.141683 (-0.055177) | 1.482434 / 1.452155 (0.030280) | 1.522773 / 1.492716 (0.030057) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.188815 / 0.018006 (0.170809) | 0.404689 / 0.000490 (0.404200) | 0.000390 / 0.000200 (0.000190) | 0.000059 / 0.000054 (0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023165 / 0.037411 (-0.014246) | 0.095934 / 0.014526 (0.081408) | 0.105788 / 0.176557 (-0.070769) | 0.169908 / 0.737135 (-0.567227) | 0.107871 / 0.296338 (-0.188467) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.457543 / 0.215209 (0.242334) | 4.563209 / 2.077655 (2.485554) | 2.172272 / 1.504120 (0.668152) | 1.965064 / 1.541195 (0.423870) | 2.020811 / 1.468490 (0.552321) | 0.705138 / 4.584777 (-3.879638) | 3.353430 / 3.745712 (-0.392283) | 1.861970 / 5.269862 (-3.407892) | 1.159201 / 4.565676 (-3.406476) | 0.083187 / 0.424275 (-0.341088) | 0.012750 / 0.007607 (0.005143) | 0.566377 / 0.226044 (0.340333) | 5.662645 / 2.268929 (3.393717) | 2.609565 / 55.444624 (-52.835059) | 2.244519 / 6.876477 (-4.631957) | 2.284111 / 2.142072 (0.142038) | 0.821974 / 4.805227 (-3.983253) | 0.151080 / 6.500664 (-6.349584) | 0.065373 / 0.075469 (-0.010096) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.230960 / 1.841788 (-0.610828) | 13.930408 / 8.074308 (5.856100) | 13.989082 / 10.191392 (3.797690) | 0.151961 / 0.680424 (-0.528462) | 0.028770 / 0.534201 (-0.505431) | 0.392269 / 0.579283 (-0.187015) | 0.400490 / 0.434364 (-0.033874) | 0.459770 / 0.540337 (-0.080568) | 0.534174 / 1.386936 (-0.852762) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006740 / 0.011353 (-0.004613) | 0.004496 / 0.011008 (-0.006512) | 0.076886 / 0.038508 (0.038377) | 0.027593 / 0.023109 (0.004484) | 0.339570 / 0.275898 (0.063672) | 0.379915 / 0.323480 (0.056435) | 0.004999 / 0.007986 (-0.002987) | 0.004253 / 0.004328 (-0.000076) | 0.074973 / 0.004250 (0.070722) | 0.037321 / 0.037052 (0.000269) | 0.344720 / 0.258489 (0.086230) | 0.398919 / 0.293841 (0.105078) | 0.032146 / 0.128546 (-0.096400) | 0.011694 / 0.075646 (-0.063952) | 0.085134 / 0.419271 (-0.334138) | 0.042328 / 0.043533 (-0.001205) | 0.339384 / 0.255139 (0.084245) | 0.368031 / 0.283200 (0.084831) | 0.092088 / 0.141683 (-0.049595) | 1.492313 / 1.452155 (0.040158) | 1.538406 / 1.492716 (0.045690) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.265619 / 0.018006 (0.247613) | 0.415478 / 0.000490 (0.414988) | 0.030221 / 0.000200 (0.030021) | 0.000277 / 0.000054 (0.000223) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024489 / 0.037411 (-0.012922) | 0.099920 / 0.014526 (0.085395) | 0.108301 / 0.176557 (-0.068256) | 0.179525 / 0.737135 (-0.557610) | 0.111492 / 0.296338 (-0.184847) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.440759 / 0.215209 (0.225550) | 4.382754 / 2.077655 (2.305100) | 2.088686 / 1.504120 (0.584566) | 1.890557 / 1.541195 (0.349363) | 1.947461 / 1.468490 (0.478971) | 0.701751 / 4.584777 (-3.883025) | 3.368896 / 3.745712 (-0.376816) | 1.867238 / 5.269862 (-3.402624) | 1.166787 / 4.565676 (-3.398890) | 0.083427 / 0.424275 (-0.340848) | 0.012406 / 0.007607 (0.004799) | 0.539467 / 0.226044 (0.313423) | 5.376083 / 2.268929 (3.107154) | 2.516566 / 55.444624 (-52.928058) | 2.177991 / 6.876477 (-4.698486) | 2.207438 / 2.142072 (0.065366) | 0.803316 / 4.805227 (-4.001911) | 0.150900 / 6.500664 (-6.349764) | 0.066328 / 0.075469 (-0.009141) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.295308 / 1.841788 (-0.546480) | 14.081343 / 8.074308 (6.007035) | 13.516853 / 10.191392 (3.325461) | 0.160530 / 0.680424 (-0.519894) | 0.016516 / 0.534201 (-0.517685) | 0.380160 / 0.579283 (-0.199123) | 0.443484 / 0.434364 (0.009120) | 0.466645 / 0.540337 (-0.073692) | 0.555339 / 1.386936 (-0.831597) |

</details>
</details>

![](https://cml.dev/watermark.png#e8a12313cd728e37b4dc4ce67864621ffc79fedb "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.011321 / 0.011353 (-0.000031) | 0.006365 / 0.011008 (-0.004643) | 0.125613 / 0.038508 (0.087105) | 0.035327 / 0.023109 (0.012218) | 0.391998 / 0.275898 (0.116100) | 0.475402 / 0.323480 (0.151923) | 0.009579 / 0.007986 (0.001593) | 0.005621 / 0.004328 (0.001293) | 0.106097 / 0.004250 (0.101846) | 0.042774 / 0.037052 (0.005722) | 0.420850 / 0.258489 (0.162361) | 0.454501 / 0.293841 (0.160660) | 0.056885 / 0.128546 (-0.071661) | 0.021718 / 0.075646 (-0.053928) | 0.419422 / 0.419271 (0.000150) | 0.056690 / 0.043533 (0.013157) | 0.405375 / 0.255139 (0.150236) | 0.444404 / 0.283200 (0.161204) | 0.136912 / 0.141683 (-0.004771) | 1.846363 / 1.452155 (0.394208) | 1.747433 / 1.492716 (0.254717) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.282260 / 0.018006 (0.264254) | 0.615813 / 0.000490 (0.615323) | 0.000515 / 0.000200 (0.000315) | 0.000106 / 0.000054 (0.000052) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029913 / 0.037411 (-0.007499) | 0.135568 / 0.014526 (0.121042) | 0.134476 / 0.176557 (-0.042081) | 0.206974 / 0.737135 (-0.530161) | 0.136976 / 0.296338 (-0.159362) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.605241 / 0.215209 (0.390032) | 6.125097 / 2.077655 (4.047442) | 2.390102 / 1.504120 (0.885982) | 2.082196 / 1.541195 (0.541001) | 2.226527 / 1.468490 (0.758037) | 1.244807 / 4.584777 (-3.339970) | 5.476437 / 3.745712 (1.730725) | 3.014970 / 5.269862 (-2.254891) | 1.963428 / 4.565676 (-2.602249) | 0.137813 / 0.424275 (-0.286462) | 0.013794 / 0.007607 (0.006187) | 0.766149 / 0.226044 (0.540104) | 7.566103 / 2.268929 (5.297175) | 3.048958 / 55.444624 (-52.395666) | 2.394819 / 6.876477 (-4.481658) | 2.416021 / 2.142072 (0.273949) | 1.369896 / 4.805227 (-3.435331) | 0.245159 / 6.500664 (-6.255506) | 0.076848 / 0.075469 (0.001379) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.530448 / 1.841788 (-0.311340) | 18.580227 / 8.074308 (10.505919) | 20.108470 / 10.191392 (9.917078) | 0.227124 / 0.680424 (-0.453300) | 0.052050 / 0.534201 (-0.482151) | 0.604565 / 0.579283 (0.025282) | 0.686475 / 0.434364 (0.252111) | 0.672298 / 0.540337 (0.131960) | 0.770552 / 1.386936 (-0.616384) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.010043 / 0.011353 (-0.001310) | 0.006445 / 0.011008 (-0.004563) | 0.099486 / 0.038508 (0.060978) | 0.037720 / 0.023109 (0.014610) | 0.425571 / 0.275898 (0.149673) | 0.467031 / 0.323480 (0.143551) | 0.007394 / 0.007986 (-0.000591) | 0.005008 / 0.004328 (0.000679) | 0.096176 / 0.004250 (0.091926) | 0.053694 / 0.037052 (0.016641) | 0.418653 / 0.258489 (0.160164) | 0.492441 / 0.293841 (0.198600) | 0.054593 / 0.128546 (-0.073953) | 0.023410 / 0.075646 (-0.052236) | 0.113825 / 0.419271 (-0.305446) | 0.066000 / 0.043533 (0.022467) | 0.418127 / 0.255139 (0.162988) | 0.457416 / 0.283200 (0.174217) | 0.119911 / 0.141683 (-0.021771) | 1.733805 / 1.452155 (0.281651) | 1.961252 / 1.492716 (0.468536) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.296126 / 0.018006 (0.278120) | 0.602169 / 0.000490 (0.601680) | 0.000454 / 0.000200 (0.000254) | 0.000075 / 0.000054 (0.000021) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032970 / 0.037411 (-0.004442) | 0.124071 / 0.014526 (0.109545) | 0.143800 / 0.176557 (-0.032757) | 0.227168 / 0.737135 (-0.509967) | 0.142817 / 0.296338 (-0.153521) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.626239 / 0.215209 (0.411030) | 6.438629 / 2.077655 (4.360974) | 2.760747 / 1.504120 (1.256627) | 2.355419 / 1.541195 (0.814224) | 2.384924 / 1.468490 (0.916434) | 1.210543 / 4.584777 (-3.374234) | 5.440389 / 3.745712 (1.694677) | 5.047939 / 5.269862 (-0.221922) | 2.759618 / 4.565676 (-1.806059) | 0.132757 / 0.424275 (-0.291518) | 0.013163 / 0.007607 (0.005556) | 0.745721 / 0.226044 (0.519677) | 7.660327 / 2.268929 (5.391398) | 3.559385 / 55.444624 (-51.885240) | 2.764344 / 6.876477 (-4.112133) | 2.975274 / 2.142072 (0.833202) | 1.460346 / 4.805227 (-3.344881) | 0.257222 / 6.500664 (-6.243443) | 0.081106 / 0.075469 (0.005637) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.698245 / 1.841788 (-0.143543) | 18.754129 / 8.074308 (10.679821) | 19.065596 / 10.191392 (8.874204) | 0.228237 / 0.680424 (-0.452187) | 0.030688 / 0.534201 (-0.503513) | 0.532561 / 0.579283 (-0.046722) | 0.601133 / 0.434364 (0.166769) | 0.620218 / 0.540337 (0.079881) | 0.751392 / 1.386936 (-0.635545) |

</details>
</details>

![](https://cml.dev/watermark.png#5f293ff23853fea210388bbef11d1621e54f22e7 "CML watermark")


(the BadZipFile error is unrelated to the changes)

<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009368 / 0.011353 (-0.001984) | 0.005143 / 0.011008 (-0.005865) | 0.100675 / 0.038508 (0.062167) | 0.036033 / 0.023109 (0.012924) | 0.297391 / 0.275898 (0.021493) | 0.362230 / 0.323480 (0.038750) | 0.008041 / 0.007986 (0.000055) | 0.004041 / 0.004328 (-0.000287) | 0.075395 / 0.004250 (0.071144) | 0.043020 / 0.037052 (0.005968) | 0.308936 / 0.258489 (0.050447) | 0.343723 / 0.293841 (0.049883) | 0.038416 / 0.128546 (-0.090131) | 0.012086 / 0.075646 (-0.063560) | 0.335102 / 0.419271 (-0.084170) | 0.047718 / 0.043533 (0.004185) | 0.297856 / 0.255139 (0.042717) | 0.317326 / 0.283200 (0.034126) | 0.101462 / 0.141683 (-0.040221) | 1.459965 / 1.452155 (0.007810) | 1.491194 / 1.492716 (-0.001522) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.211311 / 0.018006 (0.193305) | 0.443663 / 0.000490 (0.443174) | 0.003654 / 0.000200 (0.003454) | 0.000076 / 0.000054 (0.000022) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027316 / 0.037411 (-0.010095) | 0.109929 / 0.014526 (0.095403) | 0.117170 / 0.176557 (-0.059387) | 0.182494 / 0.737135 (-0.554641) | 0.124693 / 0.296338 (-0.171646) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.395904 / 0.215209 (0.180695) | 3.950906 / 2.077655 (1.873251) | 1.768807 / 1.504120 (0.264687) | 1.578979 / 1.541195 (0.037784) | 1.689976 / 1.468490 (0.221486) | 0.696458 / 4.584777 (-3.888319) | 3.750491 / 3.745712 (0.004778) | 2.117863 / 5.269862 (-3.151998) | 1.340403 / 4.565676 (-3.225274) | 0.085752 / 0.424275 (-0.338523) | 0.012206 / 0.007607 (0.004599) | 0.505561 / 0.226044 (0.279517) | 5.048721 / 2.268929 (2.779792) | 2.256623 / 55.444624 (-53.188001) | 1.905912 / 6.876477 (-4.970565) | 1.988400 / 2.142072 (-0.153672) | 0.843066 / 4.805227 (-3.962161) | 0.165717 / 6.500664 (-6.334947) | 0.062910 / 0.075469 (-0.012559) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.225668 / 1.841788 (-0.616120) | 14.660082 / 8.074308 (6.585773) | 14.295369 / 10.191392 (4.103977) | 0.171075 / 0.680424 (-0.509348) | 0.029279 / 0.534201 (-0.504922) | 0.441559 / 0.579283 (-0.137724) | 0.445382 / 0.434364 (0.011018) | 0.525350 / 0.540337 (-0.014987) | 0.608493 / 1.386936 (-0.778443) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007288 / 0.011353 (-0.004065) | 0.004999 / 0.011008 (-0.006009) | 0.074656 / 0.038508 (0.036147) | 0.033897 / 0.023109 (0.010788) | 0.345826 / 0.275898 (0.069928) | 0.390891 / 0.323480 (0.067411) | 0.005811 / 0.007986 (-0.002174) | 0.003976 / 0.004328 (-0.000353) | 0.073546 / 0.004250 (0.069295) | 0.047245 / 0.037052 (0.010193) | 0.351851 / 0.258489 (0.093362) | 0.403217 / 0.293841 (0.109376) | 0.036771 / 0.128546 (-0.091775) | 0.012240 / 0.075646 (-0.063407) | 0.086720 / 0.419271 (-0.332552) | 0.049440 / 0.043533 (0.005907) | 0.339520 / 0.255139 (0.084381) | 0.372160 / 0.283200 (0.088961) | 0.100813 / 0.141683 (-0.040870) | 1.436436 / 1.452155 (-0.015718) | 1.514723 / 1.492716 (0.022007) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.231394 / 0.018006 (0.213388) | 0.440825 / 0.000490 (0.440336) | 0.000994 / 0.000200 (0.000794) | 0.000077 / 0.000054 (0.000022) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028999 / 0.037411 (-0.008412) | 0.111391 / 0.014526 (0.096865) | 0.123058 / 0.176557 (-0.053498) | 0.194348 / 0.737135 (-0.542787) | 0.125730 / 0.296338 (-0.170609) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.431950 / 0.215209 (0.216741) | 4.298724 / 2.077655 (2.221069) | 2.064116 / 1.504120 (0.559996) | 1.892062 / 1.541195 (0.350867) | 1.985441 / 1.468490 (0.516951) | 0.707028 / 4.584777 (-3.877749) | 3.812976 / 3.745712 (0.067264) | 3.078704 / 5.269862 (-2.191158) | 1.832737 / 4.565676 (-2.732939) | 0.086182 / 0.424275 (-0.338093) | 0.012289 / 0.007607 (0.004681) | 0.530265 / 0.226044 (0.304220) | 5.283122 / 2.268929 (3.014194) | 2.558491 / 55.444624 (-52.886134) | 2.237046 / 6.876477 (-4.639431) | 2.354548 / 2.142072 (0.212475) | 0.848947 / 4.805227 (-3.956280) | 0.167907 / 6.500664 (-6.332757) | 0.064998 / 0.075469 (-0.010471) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.248287 / 1.841788 (-0.593500) | 14.976327 / 8.074308 (6.902019) | 13.596143 / 10.191392 (3.404751) | 0.145730 / 0.680424 (-0.534694) | 0.017340 / 0.534201 (-0.516861) | 0.430111 / 0.579283 (-0.149172) | 0.433462 / 0.434364 (-0.000902) | 0.540365 / 0.540337 (0.000028) | 0.650586 / 1.386936 (-0.736350) |

</details>
</details>

![](https://cml.dev/watermark.png#1875c8a4c928aeaccc826f13ffdbf7543112024d "CML watermark")