Release: 2.13.0
Opened by lhoestq (closed)


Comments:
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006498 / 0.011353 (-0.004855) | 0.003970 / 0.011008 (-0.007038) | 0.099242 / 0.038508 (0.060734) | 0.044363 / 0.023109 (0.021254) | 0.313900 / 0.275898 (0.038002) | 0.386562 / 0.323480 (0.063082) | 0.003837 / 0.007986 (-0.004149) | 0.004203 / 0.004328 (-0.000125) | 0.076191 / 0.004250 (0.071940) | 0.058823 / 0.037052 (0.021771) | 0.333838 / 0.258489 (0.075349) | 0.368235 / 0.293841 (0.074394) | 0.030774 / 0.128546 (-0.097772) | 0.008787 / 0.075646 (-0.066860) | 0.326474 / 0.419271 (-0.092798) | 0.050903 / 0.043533 (0.007370) | 0.303928 / 0.255139 (0.048789) | 0.321532 / 0.283200 (0.038333) | 0.024162 / 0.141683 (-0.117520) | 1.479662 / 1.452155 (0.027507) | 1.520300 / 1.492716 (0.027584) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.212403 / 0.018006 (0.194397) | 0.448019 / 0.000490 (0.447529) | 0.005465 / 0.000200 (0.005265) | 0.000388 / 0.000054 (0.000334) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027533 / 0.037411 (-0.009878) | 0.117477 / 0.014526 (0.102952) | 0.121182 / 0.176557 (-0.055374) | 0.181150 / 0.737135 (-0.555985) | 0.128557 / 0.296338 (-0.167782) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.397763 / 0.215209 (0.182554) | 3.959460 / 2.077655 (1.881805) | 1.822057 / 1.504120 (0.317937) | 1.627020 / 1.541195 (0.085826) | 1.695394 / 1.468490 (0.226904) | 0.536848 / 4.584777 (-4.047929) | 3.765205 / 3.745712 (0.019493) | 3.196300 / 5.269862 (-2.073561) | 1.623583 / 4.565676 (-2.942094) | 0.065823 / 0.424275 (-0.358452) | 0.011062 / 0.007607 (0.003455) | 0.500428 / 0.226044 (0.274384) | 5.008816 / 2.268929 (2.739888) | 2.314660 / 55.444624 (-53.129965) | 2.007429 / 6.876477 (-4.869047) | 2.141438 / 2.142072 (-0.000635) | 0.656697 / 4.805227 (-4.148530) | 0.143555 / 6.500664 (-6.357109) | 0.063928 / 0.075469 (-0.011541) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.169038 / 1.841788 (-0.672750) | 15.027186 / 8.074308 (6.952878) | 13.571484 / 10.191392 (3.380092) | 0.166437 / 0.680424 (-0.513986) | 0.017656 / 0.534201 (-0.516545) | 0.397725 / 0.579283 (-0.181558) | 0.451019 / 0.434364 (0.016655) | 0.469134 / 0.540337 (-0.071203) | 0.575885 / 1.386936 (-0.811051) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006887 / 0.011353 (-0.004465) | 0.004166 / 0.011008 (-0.006842) | 0.077137 / 0.038508 (0.038629) | 0.055631 / 0.023109 (0.032522) | 0.397658 / 0.275898 (0.121760) | 0.473981 / 0.323480 (0.150502) | 0.005365 / 0.007986 (-0.002621) | 0.003401 / 0.004328 (-0.000928) | 0.076481 / 0.004250 (0.072231) | 0.056014 / 0.037052 (0.018961) | 0.415253 / 0.258489 (0.156764) | 0.457620 / 0.293841 (0.163779) | 0.031850 / 0.128546 (-0.096696) | 0.008869 / 0.075646 (-0.066777) | 0.083475 / 0.419271 (-0.335796) | 0.049232 / 0.043533 (0.005699) | 0.392947 / 0.255139 (0.137808) | 0.417243 / 0.283200 (0.134043) | 0.024554 / 0.141683 (-0.117129) | 1.508081 / 1.452155 (0.055926) | 1.541845 / 1.492716 (0.049129) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.228470 / 0.018006 (0.210464) | 0.450933 / 0.000490 (0.450443) | 0.001508 / 0.000200 (0.001308) | 0.000083 / 0.000054 (0.000029) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030189 / 0.037411 (-0.007222) | 0.118853 / 0.014526 (0.104327) | 0.124809 / 0.176557 (-0.051747) | 0.175066 / 0.737135 (-0.562069) | 0.129819 / 0.296338 (-0.166519) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.451830 / 0.215209 (0.236621) | 4.505352 / 2.077655 (2.427698) | 2.309303 / 1.504120 (0.805183) | 2.120983 / 1.541195 (0.579789) | 2.198808 / 1.468490 (0.730317) | 0.543836 / 4.584777 (-4.040940) | 3.836650 / 3.745712 (0.090938) | 1.872293 / 5.269862 (-3.397568) | 1.122335 / 4.565676 (-3.443342) | 0.067463 / 0.424275 (-0.356812) | 0.012143 / 0.007607 (0.004536) | 0.553674 / 0.226044 (0.327630) | 5.572101 / 2.268929 (3.303173) | 2.772151 / 55.444624 (-52.672473) | 2.451557 / 6.876477 (-4.424920) | 2.521241 / 2.142072 (0.379169) | 0.665799 / 4.805227 (-4.139428) | 0.143842 / 6.500664 (-6.356822) | 0.065373 / 0.075469 (-0.010096) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.271013 / 1.841788 (-0.570775) | 15.290054 / 8.074308 (7.215746) | 14.807044 / 10.191392 (4.615652) | 0.163767 / 0.680424 (-0.516657) | 0.017383 / 0.534201 (-0.516818) | 0.393046 / 0.579283 (-0.186237) | 0.423056 / 0.434364 (-0.011308) | 0.459193 / 0.540337 (-0.081145) | 0.559964 / 1.386936 (-0.826972) |

</details>
</details>

![](https://cml.dev/watermark.png#011b75f044ef7fa6b8981ef3496615296aeb315b "CML watermark")


_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006112 / 0.011353 (-0.005241) | 0.003712 / 0.011008 (-0.007297) | 0.099996 / 0.038508 (0.061488) | 0.037526 / 0.023109 (0.014417) | 0.305834 / 0.275898 (0.029936) | 0.361368 / 0.323480 (0.037888) | 0.004849 / 0.007986 (-0.003136) | 0.002912 / 0.004328 (-0.001417) | 0.077729 / 0.004250 (0.073479) | 0.053203 / 0.037052 (0.016151) | 0.318088 / 0.258489 (0.059599) | 0.371745 / 0.293841 (0.077904) | 0.029384 / 0.128546 (-0.099162) | 0.008504 / 0.075646 (-0.067142) | 0.318472 / 0.419271 (-0.100799) | 0.046043 / 0.043533 (0.002510) | 0.310418 / 0.255139 (0.055279) | 0.335044 / 0.283200 (0.051844) | 0.020364 / 0.141683 (-0.121319) | 1.503201 / 1.452155 (0.051047) | 1.556408 / 1.492716 (0.063692) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.210245 / 0.018006 (0.192239) | 0.418918 / 0.000490 (0.418428) | 0.002552 / 0.000200 (0.002352) | 0.000084 / 0.000054 (0.000030) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022295 / 0.037411 (-0.015116) | 0.099534 / 0.014526 (0.085008) | 0.106432 / 0.176557 (-0.070124) | 0.165110 / 0.737135 (-0.572026) | 0.109851 / 0.296338 (-0.186488) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.423947 / 0.215209 (0.208738) | 4.232978 / 2.077655 (2.155323) | 2.004849 / 1.504120 (0.500729) | 1.814345 / 1.541195 (0.273151) | 1.809192 / 1.468490 (0.340702) | 0.561146 / 4.584777 (-4.023631) | 3.385043 / 3.745712 (-0.360669) | 1.708265 / 5.269862 (-3.561597) | 1.030290 / 4.565676 (-3.535387) | 0.067095 / 0.424275 (-0.357180) | 0.011052 / 0.007607 (0.003445) | 0.522416 / 0.226044 (0.296371) | 5.207003 / 2.268929 (2.938075) | 2.367067 / 55.444624 (-53.077558) | 1.998705 / 6.876477 (-4.877772) | 2.068633 / 2.142072 (-0.073439) | 0.672396 / 4.805227 (-4.132831) | 0.135818 / 6.500664 (-6.364846) | 0.065229 / 0.075469 (-0.010240) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.187079 / 1.841788 (-0.654709) | 13.893153 / 8.074308 (5.818845) | 13.951328 / 10.191392 (3.759936) | 0.142519 / 0.680424 (-0.537905) | 0.016546 / 0.534201 (-0.517655) | 0.364008 / 0.579283 (-0.215275) | 0.385957 / 0.434364 (-0.048407) | 0.425218 / 0.540337 (-0.115120) | 0.519586 / 1.386936 (-0.867350) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005914 / 0.011353 (-0.005439) | 0.003619 / 0.011008 (-0.007389) | 0.077806 / 0.038508 (0.039298) | 0.037254 / 0.023109 (0.014144) | 0.378976 / 0.275898 (0.103078) | 0.433620 / 0.323480 (0.110140) | 0.003291 / 0.007986 (-0.004694) | 0.004523 / 0.004328 (0.000194) | 0.077604 / 0.004250 (0.073353) | 0.047493 / 0.037052 (0.010441) | 0.396027 / 0.258489 (0.137538) | 0.453345 / 0.293841 (0.159504) | 0.028170 / 0.128546 (-0.100376) | 0.008431 / 0.075646 (-0.067215) | 0.083985 / 0.419271 (-0.335286) | 0.045149 / 0.043533 (0.001617) | 0.369364 / 0.255139 (0.114225) | 0.407191 / 0.283200 (0.123991) | 0.024033 / 0.141683 (-0.117649) | 1.516838 / 1.452155 (0.064683) | 1.564260 / 1.492716 (0.071544) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.200848 / 0.018006 (0.182842) | 0.407818 / 0.000490 (0.407328) | 0.003971 / 0.000200 (0.003771) | 0.000077 / 0.000054 (0.000023) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025033 / 0.037411 (-0.012378) | 0.103585 / 0.014526 (0.089059) | 0.108741 / 0.176557 (-0.067816) | 0.161061 / 0.737135 (-0.576075) | 0.112763 / 0.296338 (-0.183576) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.479913 / 0.215209 (0.264704) | 4.801904 / 2.077655 (2.724249) | 2.511433 / 1.504120 (1.007313) | 2.307523 / 1.541195 (0.766328) | 2.338343 / 1.468490 (0.869853) | 0.557731 / 4.584777 (-4.027046) | 3.386261 / 3.745712 (-0.359451) | 2.999978 / 5.269862 (-2.269883) | 1.463058 / 4.565676 (-3.102619) | 0.067645 / 0.424275 (-0.356630) | 0.011224 / 0.007607 (0.003617) | 0.596854 / 0.226044 (0.370810) | 5.940946 / 2.268929 (3.672017) | 2.980194 / 55.444624 (-52.464430) | 2.634961 / 6.876477 (-4.241516) | 2.648160 / 2.142072 (0.506088) | 0.669728 / 4.805227 (-4.135499) | 0.135536 / 6.500664 (-6.365128) | 0.066865 / 0.075469 (-0.008604) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.287151 / 1.841788 (-0.554637) | 14.491681 / 8.074308 (6.417373) | 14.185752 / 10.191392 (3.994360) | 0.129391 / 0.680424 (-0.551032) | 0.016650 / 0.534201 (-0.517551) | 0.380111 / 0.579283 (-0.199172) | 0.392877 / 0.434364 (-0.041487) | 0.439402 / 0.540337 (-0.100935) | 0.530865 / 1.386936 (-0.856071) |

</details>
</details>

![](https://cml.dev/watermark.png#9aaee6fd0b2bcbe18e4829602084bcd83d669c5e "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.011446 / 0.011353 (0.000093) | 0.006623 / 0.011008 (-0.004386) | 0.131915 / 0.038508 (0.093407) | 0.047364 / 0.023109 (0.024255) | 0.369203 / 0.275898 (0.093305) | 0.451509 / 0.323480 (0.128029) | 0.006265 / 0.007986 (-0.001720) | 0.004072 / 0.004328 (-0.000257) | 0.098626 / 0.004250 (0.094375) | 0.079523 / 0.037052 (0.042470) | 0.406038 / 0.258489 (0.147549) | 0.450564 / 0.293841 (0.156723) | 0.050793 / 0.128546 (-0.077753) | 0.014667 / 0.075646 (-0.060979) | 0.401359 / 0.419271 (-0.017913) | 0.072299 / 0.043533 (0.028767) | 0.404456 / 0.255139 (0.149317) | 0.396223 / 0.283200 (0.113023) | 0.037048 / 0.141683 (-0.104635) | 1.869123 / 1.452155 (0.416968) | 1.953621 / 1.492716 (0.460905) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.237246 / 0.018006 (0.219240) | 0.533207 / 0.000490 (0.532717) | 0.007392 / 0.000200 (0.007192) | 0.000117 / 0.000054 (0.000062) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029458 / 0.037411 (-0.007954) | 0.112438 / 0.014526 (0.097912) | 0.139115 / 0.176557 (-0.037441) | 0.215225 / 0.737135 (-0.521911) | 0.134440 / 0.296338 (-0.161898) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.616783 / 0.215209 (0.401574) | 6.113925 / 2.077655 (4.036270) | 2.403465 / 1.504120 (0.899345) | 1.967523 / 1.541195 (0.426329) | 2.042144 / 1.468490 (0.573654) | 0.927447 / 4.584777 (-3.657330) | 5.280413 / 3.745712 (1.534701) | 2.715335 / 5.269862 (-2.554527) | 1.755640 / 4.565676 (-2.810036) | 0.114370 / 0.424275 (-0.309905) | 0.013583 / 0.007607 (0.005976) | 0.761701 / 0.226044 (0.535657) | 7.466049 / 2.268929 (5.197120) | 3.041943 / 55.444624 (-52.402682) | 2.314477 / 6.876477 (-4.562000) | 2.469285 / 2.142072 (0.327213) | 1.216055 / 4.805227 (-3.589172) | 0.214205 / 6.500664 (-6.286459) | 0.080901 / 0.075469 (0.005432) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.565185 / 1.841788 (-0.276603) | 18.387986 / 8.074308 (10.313678) | 19.665109 / 10.191392 (9.473717) | 0.226670 / 0.680424 (-0.453754) | 0.028430 / 0.534201 (-0.505771) | 0.510526 / 0.579283 (-0.068757) | 0.623178 / 0.434364 (0.188814) | 0.592039 / 0.540337 (0.051702) | 0.728462 / 1.386936 (-0.658474) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009161 / 0.011353 (-0.002192) | 0.004891 / 0.011008 (-0.006117) | 0.106502 / 0.038508 (0.067994) | 0.048234 / 0.023109 (0.025125) | 0.451173 / 0.275898 (0.175275) | 0.557948 / 0.323480 (0.234468) | 0.005350 / 0.007986 (-0.002635) | 0.004559 / 0.004328 (0.000230) | 0.110393 / 0.004250 (0.106142) | 0.060624 / 0.037052 (0.023572) | 0.459265 / 0.258489 (0.200776) | 0.575302 / 0.293841 (0.281461) | 0.051379 / 0.128546 (-0.077167) | 0.015576 / 0.075646 (-0.060070) | 0.116650 / 0.419271 (-0.302621) | 0.065534 / 0.043533 (0.022001) | 0.461431 / 0.255139 (0.206292) | 0.487677 / 0.283200 (0.204477) | 0.037773 / 0.141683 (-0.103910) | 1.992416 / 1.452155 (0.540261) | 1.991280 / 1.492716 (0.498564) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.233607 / 0.018006 (0.215601) | 0.507539 / 0.000490 (0.507049) | 0.001307 / 0.000200 (0.001107) | 0.000096 / 0.000054 (0.000042) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032897 / 0.037411 (-0.004514) | 0.126549 / 0.014526 (0.112023) | 0.137893 / 0.176557 (-0.038663) | 0.192124 / 0.737135 (-0.545012) | 0.147300 / 0.296338 (-0.149038) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.679371 / 0.215209 (0.464162) | 6.673249 / 2.077655 (4.595595) | 2.979141 / 1.504120 (1.475022) | 2.568789 / 1.541195 (1.027594) | 2.537540 / 1.468490 (1.069050) | 0.973555 / 4.584777 (-3.611222) | 5.313536 / 3.745712 (1.567824) | 2.693283 / 5.269862 (-2.576579) | 1.819483 / 4.565676 (-2.746194) | 0.111644 / 0.424275 (-0.312631) | 0.013218 / 0.007607 (0.005611) | 0.776114 / 0.226044 (0.550070) | 7.758907 / 2.268929 (5.489978) | 3.417611 / 55.444624 (-52.027013) | 2.859502 / 6.876477 (-4.016975) | 2.927726 / 2.142072 (0.785653) | 1.163671 / 4.805227 (-3.641556) | 0.228636 / 6.500664 (-6.272028) | 0.082077 / 0.075469 (0.006607) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.746150 / 1.841788 (-0.095637) | 17.961955 / 8.074308 (9.887647) | 21.590545 / 10.191392 (11.399153) | 0.210017 / 0.680424 (-0.470406) | 0.028435 / 0.534201 (-0.505766) | 0.509253 / 0.579283 (-0.070030) | 0.606993 / 0.434364 (0.172629) | 0.587189 / 0.540337 (0.046851) | 0.684023 / 1.386936 (-0.702913) |

</details>
</details>

![](https://cml.dev/watermark.png#9aaee6fd0b2bcbe18e4829602084bcd83d669c5e "CML watermark")