Release: 2.10.1
State: closed | Author: lhoestq (MEMBER)


Comments: <details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008717 / 0.011353 (-0.002636) | 0.004570 / 0.011008 (-0.006439) | 0.100228 / 0.038508 (0.061720) | 0.030076 / 0.023109 (0.006967) | 0.317919 / 0.275898 (0.042021) | 0.366360 / 0.323480 (0.042880) | 0.007008 / 0.007986 (-0.000978) | 0.003498 / 0.004328 (-0.000831) | 0.077607 / 0.004250 (0.073356) | 0.036106 / 0.037052 (-0.000946) | 0.314128 / 0.258489 (0.055639) | 0.351450 / 0.293841 (0.057609) | 0.033697 / 0.128546 (-0.094849) | 0.011424 / 0.075646 (-0.064222) | 0.323867 / 0.419271 (-0.095404) | 0.042073 / 0.043533 (-0.001460) | 0.304564 / 0.255139 (0.049425) | 0.334865 / 0.283200 (0.051665) | 0.087791 / 0.141683 (-0.053892) | 1.488075 / 1.452155 (0.035920) | 1.513676 / 1.492716 (0.020959) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.010936 / 0.018006 (-0.007070) | 0.409610 / 0.000490 (0.409121) | 0.004820 / 0.000200 (0.004620) | 0.000079 / 0.000054 (0.000025) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023931 / 0.037411 (-0.013481) | 0.096826 / 0.014526 (0.082300) | 0.105764 / 0.176557 (-0.070792) | 0.153241 / 0.737135 (-0.583895) | 0.108976 / 0.296338 (-0.187363) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.412833 / 0.215209 (0.197624) | 4.129735 / 2.077655 (2.052081) | 1.819049 / 1.504120 (0.314929) | 1.617411 / 1.541195 (0.076216) | 1.682353 / 1.468490 (0.213863) | 0.688987 / 4.584777 (-3.895790) | 3.388276 / 3.745712 (-0.357436) | 1.857452 / 5.269862 (-3.412410) | 1.158020 / 4.565676 (-3.407657) | 0.082161 / 0.424275 (-0.342114) | 0.012319 / 0.007607 (0.004712) | 0.523052 / 0.226044 (0.297008) | 5.237726 / 2.268929 (2.968797) | 2.275605 / 55.444624 (-53.169020) | 1.931664 / 6.876477 (-4.944813) | 1.970026 / 2.142072 (-0.172046) | 0.805240 / 4.805227 (-3.999988) | 0.148431 / 6.500664 (-6.352233) | 0.064707 / 0.075469 (-0.010762) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.196456 / 1.841788 (-0.645332) | 13.750113 / 8.074308 (5.675805) | 13.853543 / 10.191392 (3.662151) | 0.137892 / 0.680424 (-0.542532) | 0.028304 / 0.534201 (-0.505897) | 0.400128 / 0.579283 (-0.179155) | 0.410409 / 0.434364 (-0.023955) | 0.479165 / 0.540337 (-0.061172) | 0.575002 / 1.386936 (-0.811934) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006587 / 0.011353 (-0.004766) | 0.004526 / 0.011008 (-0.006482) | 0.075673 / 0.038508 (0.037165) | 0.027429 / 0.023109 (0.004320) | 0.341808 / 0.275898 (0.065910) | 0.379520 / 0.323480 (0.056040) | 0.004972 / 0.007986 (-0.003014) | 0.003354 / 0.004328 (-0.000975) | 0.075373 / 0.004250 (0.071123) | 0.038347 / 0.037052 (0.001294) | 0.343671 / 0.258489 (0.085181) | 0.389632 / 0.293841 (0.095791) | 0.031694 / 0.128546 (-0.096853) | 0.011458 / 0.075646 (-0.064188) | 0.084210 / 0.419271 (-0.335062) | 0.042662 / 0.043533 (-0.000871) | 0.339436 / 0.255139 (0.084297) | 0.367493 / 0.283200 (0.084294) | 0.091604 / 0.141683 (-0.050079) | 1.526762 / 1.452155 (0.074607) | 1.569110 / 1.492716 (0.076394) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.211496 / 0.018006 (0.193489) | 0.404868 / 0.000490 (0.404379) | 0.004267 / 0.000200 (0.004067) | 0.000083 / 0.000054 (0.000029) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025189 / 0.037411 (-0.012222) | 0.099139 / 0.014526 (0.084613) | 0.105898 / 0.176557 (-0.070659) | 0.160997 / 0.737135 (-0.576138) | 0.110158 / 0.296338 (-0.186180) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.444286 / 0.215209 (0.229077) | 4.445479 / 2.077655 (2.367824) | 2.118920 / 1.504120 (0.614800) | 1.908296 / 1.541195 (0.367102) | 1.947211 / 1.468490 (0.478721) | 0.704850 / 4.584777 (-3.879927) | 3.395990 / 3.745712 (-0.349723) | 1.892529 / 5.269862 (-3.377332) | 1.172190 / 4.565676 (-3.393486) | 0.084235 / 0.424275 (-0.340040) | 0.012588 / 0.007607 (0.004981) | 0.546962 / 0.226044 (0.320918) | 5.475842 / 2.268929 (3.206913) | 2.575280 / 55.444624 (-52.869344) | 2.245658 / 6.876477 (-4.630818) | 2.274767 / 2.142072 (0.132695) | 0.813755 / 4.805227 (-3.991473) | 0.151927 / 6.500664 (-6.348737) | 0.067167 / 0.075469 (-0.008302) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.267666 / 1.841788 (-0.574122) | 13.658905 / 8.074308 (5.584597) | 13.207249 / 10.191392 (3.015857) | 0.128590 / 0.680424 (-0.551833) | 0.016531 / 0.534201 (-0.517670) | 0.385050 / 0.579283 (-0.194233) | 0.388945 / 0.434364 (-0.045419) | 0.472378 / 0.540337 (-0.067959) | 0.568929 / 1.386936 (-0.818007) |

</details>
</details>

![](https://cml.dev/watermark.png#87cd5f7f7fda60d0f91f50424bcc3f327fe0d059 "CML watermark")


_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009339 / 0.011353 (-0.002014) | 0.005197 / 0.011008 (-0.005811) | 0.100698 / 0.038508 (0.062190) | 0.035484 / 0.023109 (0.012375) | 0.299030 / 0.275898 (0.023132) | 0.366603 / 0.323480 (0.043124) | 0.007909 / 0.007986 (-0.000077) | 0.005683 / 0.004328 (0.001355) | 0.077719 / 0.004250 (0.073469) | 0.042147 / 0.037052 (0.005094) | 0.310174 / 0.258489 (0.051685) | 0.342720 / 0.293841 (0.048879) | 0.039679 / 0.128546 (-0.088867) | 0.012042 / 0.075646 (-0.063605) | 0.335663 / 0.419271 (-0.083609) | 0.051137 / 0.043533 (0.007604) | 0.298218 / 0.255139 (0.043079) | 0.316398 / 0.283200 (0.033198) | 0.108906 / 0.141683 (-0.032776) | 1.422823 / 1.452155 (-0.029331) | 1.472955 / 1.492716 (-0.019761) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.205845 / 0.018006 (0.187839) | 0.445942 / 0.000490 (0.445453) | 0.003553 / 0.000200 (0.003353) | 0.000083 / 0.000054 (0.000028) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025506 / 0.037411 (-0.011906) | 0.107494 / 0.014526 (0.092969) | 0.116226 / 0.176557 (-0.060331) | 0.157313 / 0.737135 (-0.579822) | 0.123822 / 0.296338 (-0.172516) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.400908 / 0.215209 (0.185699) | 3.980232 / 2.077655 (1.902578) | 1.805410 / 1.504120 (0.301290) | 1.615698 / 1.541195 (0.074503) | 1.677213 / 1.468490 (0.208723) | 0.697882 / 4.584777 (-3.886895) | 3.752781 / 3.745712 (0.007069) | 2.076062 / 5.269862 (-3.193800) | 1.446909 / 4.565676 (-3.118768) | 0.084572 / 0.424275 (-0.339703) | 0.011917 / 0.007607 (0.004310) | 0.511815 / 0.226044 (0.285771) | 5.121487 / 2.268929 (2.852558) | 2.277642 / 55.444624 (-53.166982) | 1.930393 / 6.876477 (-4.946084) | 1.965855 / 2.142072 (-0.176218) | 0.843391 / 4.805227 (-3.961837) | 0.163581 / 6.500664 (-6.337083) | 0.062547 / 0.075469 (-0.012922) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.223930 / 1.841788 (-0.617858) | 14.354466 / 8.074308 (6.280158) | 14.015159 / 10.191392 (3.823767) | 0.148658 / 0.680424 (-0.531766) | 0.028469 / 0.534201 (-0.505732) | 0.437614 / 0.579283 (-0.141669) | 0.435452 / 0.434364 (0.001089) | 0.523623 / 0.540337 (-0.016715) | 0.625109 / 1.386936 (-0.761827) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006917 / 0.011353 (-0.004436) | 0.005080 / 0.011008 (-0.005928) | 0.075806 / 0.038508 (0.037298) | 0.032402 / 0.023109 (0.009293) | 0.331105 / 0.275898 (0.055207) | 0.361226 / 0.323480 (0.037746) | 0.005694 / 0.007986 (-0.002292) | 0.003810 / 0.004328 (-0.000518) | 0.076886 / 0.004250 (0.072635) | 0.046158 / 0.037052 (0.009106) | 0.338791 / 0.258489 (0.080302) | 0.385733 / 0.293841 (0.091892) | 0.035590 / 0.128546 (-0.092956) | 0.011997 / 0.075646 (-0.063649) | 0.087854 / 0.419271 (-0.331417) | 0.048985 / 0.043533 (0.005452) | 0.331248 / 0.255139 (0.076109) | 0.354633 / 0.283200 (0.071434) | 0.101609 / 0.141683 (-0.040074) | 1.496899 / 1.452155 (0.044745) | 1.570469 / 1.492716 (0.077753) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.180871 / 0.018006 (0.162865) | 0.449417 / 0.000490 (0.448928) | 0.004300 / 0.000200 (0.004100) | 0.000102 / 0.000054 (0.000048) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029054 / 0.037411 (-0.008358) | 0.110888 / 0.014526 (0.096362) | 0.121736 / 0.176557 (-0.054821) | 0.172563 / 0.737135 (-0.564572) | 0.126565 / 0.296338 (-0.169773) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.419545 / 0.215209 (0.204336) | 4.193685 / 2.077655 (2.116031) | 2.049967 / 1.504120 (0.545847) | 1.855038 / 1.541195 (0.313843) | 1.899822 / 1.468490 (0.431332) | 0.709123 / 4.584777 (-3.875654) | 3.795939 / 3.745712 (0.050227) | 2.076055 / 5.269862 (-3.193807) | 1.335864 / 4.565676 (-3.229812) | 0.085555 / 0.424275 (-0.338720) | 0.012197 / 0.007607 (0.004590) | 0.516164 / 0.226044 (0.290119) | 5.158983 / 2.268929 (2.890054) | 2.445581 / 55.444624 (-52.999044) | 2.122256 / 6.876477 (-4.754221) | 2.160011 / 2.142072 (0.017939) | 0.840251 / 4.805227 (-3.964976) | 0.165924 / 6.500664 (-6.334740) | 0.064080 / 0.075469 (-0.011389) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.285292 / 1.841788 (-0.556495) | 14.561084 / 8.074308 (6.486776) | 12.899269 / 10.191392 (2.707877) | 0.185657 / 0.680424 (-0.494767) | 0.017866 / 0.534201 (-0.516335) | 0.425365 / 0.579283 (-0.153918) | 0.427183 / 0.434364 (-0.007181) | 0.529773 / 0.540337 (-0.010564) | 0.642061 / 1.386936 (-0.744875) |

</details>
</details>

![](https://cml.dev/watermark.png#0628013d009dd5150e8a1c1a4ac9d93887b88a76 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008995 / 0.011353 (-0.002357) | 0.004540 / 0.011008 (-0.006469) | 0.099675 / 0.038508 (0.061167) | 0.030338 / 0.023109 (0.007229) | 0.307167 / 0.275898 (0.031269) | 0.338789 / 0.323480 (0.015309) | 0.007293 / 0.007986 (-0.000692) | 0.004681 / 0.004328 (0.000352) | 0.077475 / 0.004250 (0.073225) | 0.036399 / 0.037052 (-0.000654) | 0.304615 / 0.258489 (0.046126) | 0.351611 / 0.293841 (0.057770) | 0.034449 / 0.128546 (-0.094097) | 0.011565 / 0.075646 (-0.064082) | 0.322765 / 0.419271 (-0.096506) | 0.041971 / 0.043533 (-0.001562) | 0.307492 / 0.255139 (0.052354) | 0.327240 / 0.283200 (0.044040) | 0.087110 / 0.141683 (-0.054573) | 1.484600 / 1.452155 (0.032445) | 1.536651 / 1.492716 (0.043934) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.185876 / 0.018006 (0.167869) | 0.404276 / 0.000490 (0.403787) | 0.001592 / 0.000200 (0.001392) | 0.000072 / 0.000054 (0.000018) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023272 / 0.037411 (-0.014139) | 0.096273 / 0.014526 (0.081747) | 0.105400 / 0.176557 (-0.071157) | 0.149720 / 0.737135 (-0.587416) | 0.107807 / 0.296338 (-0.188532) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.420072 / 0.215209 (0.204863) | 4.184108 / 2.077655 (2.106454) | 1.880690 / 1.504120 (0.376570) | 1.673103 / 1.541195 (0.131909) | 1.715792 / 1.468490 (0.247302) | 0.695771 / 4.584777 (-3.889006) | 3.450224 / 3.745712 (-0.295488) | 2.999218 / 5.269862 (-2.270644) | 1.585571 / 4.565676 (-2.980106) | 0.082105 / 0.424275 (-0.342170) | 0.012453 / 0.007607 (0.004846) | 0.528538 / 0.226044 (0.302494) | 5.287951 / 2.268929 (3.019023) | 2.289127 / 55.444624 (-53.155497) | 1.956503 / 6.876477 (-4.919974) | 2.004498 / 2.142072 (-0.137575) | 0.813547 / 4.805227 (-3.991681) | 0.151574 / 6.500664 (-6.349090) | 0.063763 / 0.075469 (-0.011706) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.239125 / 1.841788 (-0.602662) | 13.627676 / 8.074308 (5.553368) | 13.747815 / 10.191392 (3.556423) | 0.157745 / 0.680424 (-0.522679) | 0.028590 / 0.534201 (-0.505611) | 0.397472 / 0.579283 (-0.181811) | 0.405925 / 0.434364 (-0.028439) | 0.477942 / 0.540337 (-0.062396) | 0.572379 / 1.386936 (-0.814557) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006637 / 0.011353 (-0.004716) | 0.004657 / 0.011008 (-0.006351) | 0.082056 / 0.038508 (0.043548) | 0.027974 / 0.023109 (0.004865) | 0.342887 / 0.275898 (0.066989) | 0.375938 / 0.323480 (0.052458) | 0.004958 / 0.007986 (-0.003028) | 0.004738 / 0.004328 (0.000409) | 0.080449 / 0.004250 (0.076198) | 0.038138 / 0.037052 (0.001085) | 0.345636 / 0.258489 (0.087147) | 0.385992 / 0.293841 (0.092151) | 0.033265 / 0.128546 (-0.095281) | 0.011965 / 0.075646 (-0.063681) | 0.091441 / 0.419271 (-0.327830) | 0.051407 / 0.043533 (0.007874) | 0.353758 / 0.255139 (0.098619) | 0.372118 / 0.283200 (0.088919) | 0.093947 / 0.141683 (-0.047735) | 1.468197 / 1.452155 (0.016042) | 1.554677 / 1.492716 (0.061960) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.222034 / 0.018006 (0.204027) | 0.403658 / 0.000490 (0.403169) | 0.003242 / 0.000200 (0.003042) | 0.000082 / 0.000054 (0.000027) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025335 / 0.037411 (-0.012076) | 0.100404 / 0.014526 (0.085878) | 0.107858 / 0.176557 (-0.068698) | 0.156115 / 0.737135 (-0.581021) | 0.113967 / 0.296338 (-0.182372) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.437567 / 0.215209 (0.222358) | 4.362486 / 2.077655 (2.284832) | 2.067315 / 1.504120 (0.563195) | 1.857669 / 1.541195 (0.316475) | 1.926380 / 1.468490 (0.457890) | 0.703905 / 4.584777 (-3.880872) | 3.437139 / 3.745712 (-0.308573) | 3.051931 / 5.269862 (-2.217930) | 1.356494 / 4.565676 (-3.209182) | 0.083679 / 0.424275 (-0.340596) | 0.012507 / 0.007607 (0.004900) | 0.539572 / 0.226044 (0.313528) | 5.405790 / 2.268929 (3.136861) | 2.532769 / 55.444624 (-52.911855) | 2.181950 / 6.876477 (-4.694527) | 2.212627 / 2.142072 (0.070554) | 0.807468 / 4.805227 (-3.997759) | 0.152146 / 6.500664 (-6.348518) | 0.068891 / 0.075469 (-0.006578) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.286972 / 1.841788 (-0.554816) | 13.987186 / 8.074308 (5.912878) | 13.115065 / 10.191392 (2.923673) | 0.162143 / 0.680424 (-0.518281) | 0.016767 / 0.534201 (-0.517434) | 0.384766 / 0.579283 (-0.194517) | 0.397438 / 0.434364 (-0.036926) | 0.470850 / 0.540337 (-0.069487) | 0.562216 / 1.386936 (-0.824720) |

</details>
</details>

![](https://cml.dev/watermark.png#2843fceabc428932754ba497f643d6e94173b91e "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.010877 / 0.011353 (-0.000476) | 0.005739 / 0.011008 (-0.005269) | 0.118542 / 0.038508 (0.080034) | 0.042266 / 0.023109 (0.019157) | 0.359317 / 0.275898 (0.083419) | 0.412995 / 0.323480 (0.089515) | 0.009158 / 0.007986 (0.001173) | 0.006343 / 0.004328 (0.002014) | 0.089587 / 0.004250 (0.085336) | 0.047899 / 0.037052 (0.010847) | 0.358745 / 0.258489 (0.100256) | 0.421316 / 0.293841 (0.127476) | 0.044540 / 0.128546 (-0.084006) | 0.013872 / 0.075646 (-0.061774) | 0.399856 / 0.419271 (-0.019415) | 0.056484 / 0.043533 (0.012951) | 0.356922 / 0.255139 (0.101783) | 0.385598 / 0.283200 (0.102398) | 0.116039 / 0.141683 (-0.025644) | 1.726095 / 1.452155 (0.273940) | 1.888643 / 1.492716 (0.395927) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.269517 / 0.018006 (0.251511) | 0.511204 / 0.000490 (0.510714) | 0.001906 / 0.000200 (0.001706) | 0.000103 / 0.000054 (0.000048) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031133 / 0.037411 (-0.006278) | 0.128513 / 0.014526 (0.113987) | 0.139639 / 0.176557 (-0.036918) | 0.189778 / 0.737135 (-0.547358) | 0.145219 / 0.296338 (-0.151120) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.486693 / 0.215209 (0.271484) | 4.851999 / 2.077655 (2.774344) | 2.255334 / 1.504120 (0.751214) | 2.052271 / 1.541195 (0.511077) | 2.143262 / 1.468490 (0.674772) | 0.835765 / 4.584777 (-3.749012) | 4.451280 / 3.745712 (0.705568) | 2.534392 / 5.269862 (-2.735469) | 1.747817 / 4.565676 (-2.817859) | 0.101186 / 0.424275 (-0.323089) | 0.014281 / 0.007607 (0.006674) | 0.616164 / 0.226044 (0.390120) | 6.161789 / 2.268929 (3.892860) | 2.815347 / 55.444624 (-52.629277) | 2.408305 / 6.876477 (-4.468172) | 2.508240 / 2.142072 (0.366167) | 1.017709 / 4.805227 (-3.787519) | 0.198272 / 6.500664 (-6.302392) | 0.075663 / 0.075469 (0.000194) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.435501 / 1.841788 (-0.406287) | 18.149581 / 8.074308 (10.075273) | 16.619011 / 10.191392 (6.427619) | 0.205080 / 0.680424 (-0.475344) | 0.033780 / 0.534201 (-0.500421) | 0.515768 / 0.579283 (-0.063515) | 0.542628 / 0.434364 (0.108264) | 0.634067 / 0.540337 (0.093730) | 0.757841 / 1.386936 (-0.629095) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008541 / 0.011353 (-0.002812) | 0.005733 / 0.011008 (-0.005275) | 0.089859 / 0.038508 (0.051351) | 0.039379 / 0.023109 (0.016270) | 0.402037 / 0.275898 (0.126139) | 0.454046 / 0.323480 (0.130566) | 0.006652 / 0.007986 (-0.001334) | 0.004555 / 0.004328 (0.000227) | 0.087651 / 0.004250 (0.083401) | 0.054934 / 0.037052 (0.017881) | 0.404468 / 0.258489 (0.145979) | 0.467127 / 0.293841 (0.173286) | 0.042034 / 0.128546 (-0.086512) | 0.014225 / 0.075646 (-0.061421) | 0.103281 / 0.419271 (-0.315990) | 0.057767 / 0.043533 (0.014234) | 0.396391 / 0.255139 (0.141252) | 0.429364 / 0.283200 (0.146165) | 0.120193 / 0.141683 (-0.021489) | 1.794029 / 1.452155 (0.341875) | 1.875431 / 1.492716 (0.382714) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.325707 / 0.018006 (0.307701) | 0.503841 / 0.000490 (0.503351) | 0.010224 / 0.000200 (0.010024) | 0.000137 / 0.000054 (0.000082) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035289 / 0.037411 (-0.002123) | 0.139018 / 0.014526 (0.124492) | 0.145112 / 0.176557 (-0.031445) | 0.202616 / 0.737135 (-0.534519) | 0.152975 / 0.296338 (-0.143363) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.493110 / 0.215209 (0.277901) | 4.885713 / 2.077655 (2.808058) | 2.344417 / 1.504120 (0.840297) | 2.135734 / 1.541195 (0.594540) | 2.254118 / 1.468490 (0.785628) | 0.811516 / 4.584777 (-3.773261) | 4.484454 / 3.745712 (0.738742) | 2.459913 / 5.269862 (-2.809948) | 1.553106 / 4.565676 (-3.012570) | 0.100943 / 0.424275 (-0.323332) | 0.014848 / 0.007607 (0.007241) | 0.626214 / 0.226044 (0.400170) | 6.206925 / 2.268929 (3.937997) | 2.986549 / 55.444624 (-52.458076) | 2.521895 / 6.876477 (-4.354582) | 2.610917 / 2.142072 (0.468845) | 0.998496 / 4.805227 (-3.806731) | 0.199405 / 6.500664 (-6.301260) | 0.077355 / 0.075469 (0.001886) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.525135 / 1.841788 (-0.316653) | 18.708407 / 8.074308 (10.634099) | 16.049482 / 10.191392 (5.858090) | 0.170986 / 0.680424 (-0.509437) | 0.021090 / 0.534201 (-0.513111) | 0.511734 / 0.579283 (-0.067549) | 0.495507 / 0.434364 (0.061143) | 0.628578 / 0.540337 (0.088241) | 0.749546 / 1.386936 (-0.637390) |

</details>
</details>

![](https://cml.dev/watermark.png#2843fceabc428932754ba497f643d6e94173b91e "CML watermark")