Issue: Release: 2.15.0
State: closed
Author: albertvillanova


Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004331 / 0.011353 (-0.007022) | 0.002573 / 0.011008 (-0.008435) | 0.061002 / 0.038508 (0.022494) | 0.029259 / 0.023109 (0.006149) | 0.242983 / 0.275898 (-0.032915) | 0.267629 / 0.323480 (-0.055851) | 0.003906 / 0.007986 (-0.004080) | 0.002383 / 0.004328 (-0.001946) | 0.047574 / 0.004250 (0.043323) | 0.042153 / 0.037052 (0.005101) | 0.245821 / 0.258489 (-0.012668) | 0.276479 / 0.293841 (-0.017362) | 0.022498 / 0.128546 (-0.106049) | 0.006775 / 0.075646 (-0.068871) | 0.201795 / 0.419271 (-0.217477) | 0.052443 / 0.043533 (0.008910) | 0.248320 / 0.255139 (-0.006819) | 0.261964 / 0.283200 (-0.021235) | 0.016764 / 0.141683 (-0.124919) | 1.118702 / 1.452155 (-0.333453) | 1.203079 / 1.492716 (-0.289638) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.088808 / 0.018006 (0.070801) | 0.296526 / 0.000490 (0.296037) | 0.000203 / 0.000200 (0.000003) | 0.000050 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018816 / 0.037411 (-0.018595) | 0.062295 / 0.014526 (0.047769) | 0.075228 / 0.176557 (-0.101329) | 0.119916 / 0.737135 (-0.617219) | 0.077206 / 0.296338 (-0.219132) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.276723 / 0.215209 (0.061514) | 2.711431 / 2.077655 (0.633776) | 1.425590 / 1.504120 (-0.078530) | 1.301383 / 1.541195 (-0.239812) | 1.316314 / 1.468490 (-0.152176) | 0.402709 / 4.584777 (-4.182068) | 2.347229 / 3.745712 (-1.398483) | 2.596937 / 5.269862 (-2.672925) | 1.560658 / 4.565676 (-3.005018) | 0.046162 / 0.424275 (-0.378113) | 0.004760 / 0.007607 (-0.002848) | 0.330522 / 0.226044 (0.104478) | 3.244072 / 2.268929 (0.975143) | 1.747603 / 55.444624 (-53.697021) | 1.475534 / 6.876477 (-5.400943) | 1.485135 / 2.142072 (-0.656938) | 0.476794 / 4.805227 (-4.328433) | 0.098496 / 6.500664 (-6.402168) | 0.040740 / 0.075469 (-0.034729) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.939020 / 1.841788 (-0.902768) | 11.235187 / 8.074308 (3.160878) | 10.194975 / 10.191392 (0.003583) | 0.126241 / 0.680424 (-0.554182) | 0.013990 / 0.534201 (-0.520211) | 0.269149 / 0.579283 (-0.310134) | 0.256950 / 0.434364 (-0.177414) | 0.301282 / 0.540337 (-0.239056) | 0.421490 / 1.386936 (-0.965446) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004956 / 0.011353 (-0.006397) | 0.002478 / 0.011008 (-0.008530) | 0.047773 / 0.038508 (0.009265) | 0.050076 / 0.023109 (0.026967) | 0.261915 / 0.275898 (-0.013983) | 0.282553 / 0.323480 (-0.040927) | 0.003881 / 0.007986 (-0.004105) | 0.002329 / 0.004328 (-0.001999) | 0.048091 / 0.004250 (0.043841) | 0.038188 / 0.037052 (0.001135) | 0.265502 / 0.258489 (0.007013) | 0.292568 / 0.293841 (-0.001273) | 0.024172 / 0.128546 (-0.104374) | 0.006865 / 0.075646 (-0.068781) | 0.053199 / 0.419271 (-0.366072) | 0.032201 / 0.043533 (-0.011332) | 0.265774 / 0.255139 (0.010635) | 0.277954 / 0.283200 (-0.005245) | 0.017798 / 0.141683 (-0.123885) | 1.121503 / 1.452155 (-0.330652) | 1.176319 / 1.492716 (-0.316398) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.087027 / 0.018006 (0.069020) | 0.296182 / 0.000490 (0.295693) | 0.000216 / 0.000200 (0.000017) | 0.000050 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.020990 / 0.037411 (-0.016421) | 0.069693 / 0.014526 (0.055168) | 0.081098 / 0.176557 (-0.095459) | 0.117760 / 0.737135 (-0.619375) | 0.081493 / 0.296338 (-0.214845) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.295078 / 0.215209 (0.079869) | 2.876602 / 2.077655 (0.798947) | 1.558011 / 1.504120 (0.053891) | 1.426715 / 1.541195 (-0.114480) | 1.443785 / 1.468490 (-0.024705) | 0.400826 / 4.584777 (-4.183951) | 2.378903 / 3.745712 (-1.366810) | 2.473128 / 5.269862 (-2.796734) | 1.500785 / 4.565676 (-3.064891) | 0.045438 / 0.424275 (-0.378837) | 0.004953 / 0.007607 (-0.002654) | 0.348182 / 0.226044 (0.122137) | 3.427751 / 2.268929 (1.158822) | 1.925173 / 55.444624 (-53.519451) | 1.633354 / 6.876477 (-5.243123) | 1.651573 / 2.142072 (-0.490499) | 0.473260 / 4.805227 (-4.331968) | 0.097613 / 6.500664 (-6.403051) | 0.040196 / 0.075469 (-0.035273) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.951780 / 1.841788 (-0.890008) | 11.709342 / 8.074308 (3.635034) | 10.571831 / 10.191392 (0.380439) | 0.134344 / 0.680424 (-0.546079) | 0.022116 / 0.534201 (-0.512084) | 0.269651 / 0.579283 (-0.309632) | 0.272310 / 0.434364 (-0.162054) | 0.306434 / 0.540337 (-0.233903) | 0.408320 / 1.386936 (-0.978616) |

</details>
</details>

![](https://cml.dev/watermark.png#7ea64b77079cf76675421917472c05d06ace63fc "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004402 / 0.011353 (-0.006951) | 0.002732 / 0.011008 (-0.008277) | 0.062799 / 0.038508 (0.024291) | 0.029155 / 0.023109 (0.006046) | 0.241925 / 0.275898 (-0.033973) | 0.275694 / 0.323480 (-0.047786) | 0.003989 / 0.007986 (-0.003997) | 0.002528 / 0.004328 (-0.001801) | 0.048410 / 0.004250 (0.044160) | 0.043729 / 0.037052 (0.006677) | 0.248843 / 0.258489 (-0.009646) | 0.282980 / 0.293841 (-0.010860) | 0.023828 / 0.128546 (-0.104718) | 0.006972 / 0.075646 (-0.068675) | 0.213222 / 0.419271 (-0.206049) | 0.054883 / 0.043533 (0.011350) | 0.251353 / 0.255139 (-0.003786) | 0.269818 / 0.283200 (-0.013381) | 0.016906 / 0.141683 (-0.124777) | 1.114109 / 1.452155 (-0.338045) | 1.162942 / 1.492716 (-0.329774) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093724 / 0.018006 (0.075718) | 0.301989 / 0.000490 (0.301499) | 0.000213 / 0.000200 (0.000014) | 0.000049 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018245 / 0.037411 (-0.019166) | 0.062237 / 0.014526 (0.047712) | 0.075644 / 0.176557 (-0.100913) | 0.119655 / 0.737135 (-0.617480) | 0.074525 / 0.296338 (-0.221814) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.274534 / 0.215209 (0.059324) | 2.683678 / 2.077655 (0.606024) | 1.453306 / 1.504120 (-0.050814) | 1.347630 / 1.541195 (-0.193564) | 1.352875 / 1.468490 (-0.115615) | 0.398425 / 4.584777 (-4.186352) | 2.375738 / 3.745712 (-1.369974) | 2.591573 / 5.269862 (-2.678289) | 1.555527 / 4.565676 (-3.010150) | 0.045656 / 0.424275 (-0.378619) | 0.004898 / 0.007607 (-0.002709) | 0.330591 / 0.226044 (0.104547) | 3.247638 / 2.268929 (0.978710) | 1.816676 / 55.444624 (-53.627948) | 1.531754 / 6.876477 (-5.344723) | 1.543196 / 2.142072 (-0.598877) | 0.472489 / 4.805227 (-4.332739) | 0.099311 / 6.500664 (-6.401353) | 0.042139 / 0.075469 (-0.033330) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.945472 / 1.841788 (-0.896316) | 11.476550 / 8.074308 (3.402242) | 10.281157 / 10.191392 (0.089765) | 0.141062 / 0.680424 (-0.539362) | 0.013634 / 0.534201 (-0.520567) | 0.268778 / 0.579283 (-0.310505) | 0.263542 / 0.434364 (-0.170822) | 0.307918 / 0.540337 (-0.232420) | 0.421231 / 1.386936 (-0.965705) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005090 / 0.011353 (-0.006263) | 0.003135 / 0.011008 (-0.007873) | 0.048058 / 0.038508 (0.009550) | 0.052898 / 0.023109 (0.029789) | 0.273233 / 0.275898 (-0.002665) | 0.299516 / 0.323480 (-0.023964) | 0.004126 / 0.007986 (-0.003860) | 0.002331 / 0.004328 (-0.001997) | 0.047627 / 0.004250 (0.043376) | 0.039076 / 0.037052 (0.002023) | 0.276625 / 0.258489 (0.018136) | 0.308180 / 0.293841 (0.014340) | 0.024929 / 0.128546 (-0.103618) | 0.007396 / 0.075646 (-0.068251) | 0.053408 / 0.419271 (-0.365863) | 0.032896 / 0.043533 (-0.010637) | 0.275412 / 0.255139 (0.020273) | 0.292014 / 0.283200 (0.008814) | 0.018336 / 0.141683 (-0.123347) | 1.123565 / 1.452155 (-0.328589) | 1.175382 / 1.492716 (-0.317334) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093799 / 0.018006 (0.075793) | 0.304219 / 0.000490 (0.303729) | 0.000231 / 0.000200 (0.000031) | 0.000050 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021034 / 0.037411 (-0.016377) | 0.069961 / 0.014526 (0.055435) | 0.080311 / 0.176557 (-0.096246) | 0.118603 / 0.737135 (-0.618532) | 0.084003 / 0.296338 (-0.212335) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.305610 / 0.215209 (0.090401) | 2.962027 / 2.077655 (0.884372) | 1.598604 / 1.504120 (0.094484) | 1.476227 / 1.541195 (-0.064967) | 1.528960 / 1.468490 (0.060470) | 0.404545 / 4.584777 (-4.180232) | 2.423147 / 3.745712 (-1.322565) | 2.516632 / 5.269862 (-2.753229) | 1.529000 / 4.565676 (-3.036677) | 0.045780 / 0.424275 (-0.378495) | 0.004784 / 0.007607 (-0.002823) | 0.358836 / 0.226044 (0.132792) | 3.508782 / 2.268929 (1.239853) | 1.954513 / 55.444624 (-53.490111) | 1.672824 / 6.876477 (-5.203653) | 1.683482 / 2.142072 (-0.458590) | 0.479014 / 4.805227 (-4.326213) | 0.098325 / 6.500664 (-6.402340) | 0.040934 / 0.075469 (-0.034536) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.974770 / 1.841788 (-0.867017) | 11.956137 / 8.074308 (3.881829) | 10.956458 / 10.191392 (0.765066) | 0.141800 / 0.680424 (-0.538624) | 0.015439 / 0.534201 (-0.518762) | 0.271783 / 0.579283 (-0.307500) | 0.278058 / 0.434364 (-0.156306) | 0.305823 / 0.540337 (-0.234514) | 0.415677 / 1.386936 (-0.971259) |

</details>
</details>

![](https://cml.dev/watermark.png#0caf91285116ec910f409e82cc6e1f4cff7496e3 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004483 / 0.011353 (-0.006870) | 0.002560 / 0.011008 (-0.008448) | 0.061428 / 0.038508 (0.022920) | 0.029460 / 0.023109 (0.006351) | 0.238971 / 0.275898 (-0.036927) | 0.271768 / 0.323480 (-0.051712) | 0.003970 / 0.007986 (-0.004016) | 0.002408 / 0.004328 (-0.001921) | 0.047755 / 0.004250 (0.043505) | 0.043358 / 0.037052 (0.006306) | 0.245543 / 0.258489 (-0.012946) | 0.278230 / 0.293841 (-0.015611) | 0.023819 / 0.128546 (-0.104727) | 0.006856 / 0.075646 (-0.068790) | 0.204603 / 0.419271 (-0.214668) | 0.054521 / 0.043533 (0.010989) | 0.246277 / 0.255139 (-0.008862) | 0.271230 / 0.283200 (-0.011969) | 0.017283 / 0.141683 (-0.124400) | 1.088955 / 1.452155 (-0.363200) | 1.245141 / 1.492716 (-0.247575) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091534 / 0.018006 (0.073528) | 0.299517 / 0.000490 (0.299027) | 0.000215 / 0.000200 (0.000015) | 0.000043 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018105 / 0.037411 (-0.019306) | 0.061860 / 0.014526 (0.047334) | 0.074494 / 0.176557 (-0.102063) | 0.120107 / 0.737135 (-0.617029) | 0.073406 / 0.296338 (-0.222932) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.278140 / 0.215209 (0.062931) | 2.746208 / 2.077655 (0.668553) | 1.476264 / 1.504120 (-0.027856) | 1.356498 / 1.541195 (-0.184697) | 1.362998 / 1.468490 (-0.105492) | 0.401884 / 4.584777 (-4.182893) | 2.409836 / 3.745712 (-1.335877) | 2.579087 / 5.269862 (-2.690775) | 1.545021 / 4.565676 (-3.020656) | 0.046001 / 0.424275 (-0.378274) | 0.004812 / 0.007607 (-0.002795) | 0.339767 / 0.226044 (0.113722) | 3.341599 / 2.268929 (1.072670) | 1.821498 / 55.444624 (-53.623127) | 1.559311 / 6.876477 (-5.317166) | 1.570368 / 2.142072 (-0.571704) | 0.472688 / 4.805227 (-4.332539) | 0.099549 / 6.500664 (-6.401115) | 0.041644 / 0.075469 (-0.033825) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.951988 / 1.841788 (-0.889799) | 11.371459 / 8.074308 (3.297150) | 10.229446 / 10.191392 (0.038054) | 0.128105 / 0.680424 (-0.552319) | 0.014418 / 0.534201 (-0.519783) | 0.268615 / 0.579283 (-0.310668) | 0.263956 / 0.434364 (-0.170407) | 0.302213 / 0.540337 (-0.238125) | 0.427224 / 1.386936 (-0.959712) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005150 / 0.011353 (-0.006203) | 0.002557 / 0.011008 (-0.008451) | 0.048092 / 0.038508 (0.009584) | 0.050522 / 0.023109 (0.027413) | 0.272195 / 0.275898 (-0.003703) | 0.294191 / 0.323480 (-0.029289) | 0.004098 / 0.007986 (-0.003887) | 0.002350 / 0.004328 (-0.001978) | 0.048682 / 0.004250 (0.044432) | 0.038381 / 0.037052 (0.001328) | 0.275530 / 0.258489 (0.017041) | 0.303991 / 0.293841 (0.010150) | 0.024734 / 0.128546 (-0.103812) | 0.006926 / 0.075646 (-0.068720) | 0.053683 / 0.419271 (-0.365588) | 0.032675 / 0.043533 (-0.010858) | 0.272816 / 0.255139 (0.017677) | 0.291754 / 0.283200 (0.008554) | 0.018290 / 0.141683 (-0.123392) | 1.127696 / 1.452155 (-0.324459) | 1.187080 / 1.492716 (-0.305636) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091224 / 0.018006 (0.073218) | 0.288838 / 0.000490 (0.288348) | 0.000226 / 0.000200 (0.000026) | 0.000045 / 0.000054 (-0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021409 / 0.037411 (-0.016003) | 0.069846 / 0.014526 (0.055320) | 0.079962 / 0.176557 (-0.096594) | 0.118575 / 0.737135 (-0.618560) | 0.080223 / 0.296338 (-0.216115) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.290835 / 0.215209 (0.075626) | 2.831787 / 2.077655 (0.754133) | 1.587728 / 1.504120 (0.083608) | 1.461939 / 1.541195 (-0.079256) | 1.495257 / 1.468490 (0.026767) | 0.397653 / 4.584777 (-4.187124) | 2.399903 / 3.745712 (-1.345809) | 2.527615 / 5.269862 (-2.742247) | 1.501555 / 4.565676 (-3.064121) | 0.045742 / 0.424275 (-0.378533) | 0.004797 / 0.007607 (-0.002811) | 0.339139 / 0.226044 (0.113094) | 3.358340 / 2.268929 (1.089412) | 1.968955 / 55.444624 (-53.475670) | 1.663598 / 6.876477 (-5.212879) | 1.673995 / 2.142072 (-0.468078) | 0.463444 / 4.805227 (-4.341783) | 0.098008 / 6.500664 (-6.402656) | 0.040836 / 0.075469 (-0.034633) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.974033 / 1.841788 (-0.867755) | 11.863206 / 8.074308 (3.788897) | 10.892389 / 10.191392 (0.700997) | 0.128884 / 0.680424 (-0.551540) | 0.015319 / 0.534201 (-0.518882) | 0.268931 / 0.579283 (-0.310353) | 0.274148 / 0.434364 (-0.160216) | 0.305407 / 0.540337 (-0.234930) | 0.410574 / 1.386936 (-0.976362) |

</details>
</details>

![](https://cml.dev/watermark.png#0caf91285116ec910f409e82cc6e1f4cff7496e3 "CML watermark")