Release: 2.13.2
Opened by albertvillanova (closed)


Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007801 / 0.011353 (-0.003552) | 0.004831 / 0.011008 (-0.006177) | 0.123101 / 0.038508 (0.084593) | 0.053246 / 0.023109 (0.030137) | 0.381787 / 0.275898 (0.105889) | 0.461822 / 0.323480 (0.138342) | 0.004655 / 0.007986 (-0.003331) | 0.004818 / 0.004328 (0.000490) | 0.090865 / 0.004250 (0.086614) | 0.070626 / 0.037052 (0.033574) | 0.409122 / 0.258489 (0.150633) | 0.449627 / 0.293841 (0.155787) | 0.037477 / 0.128546 (-0.091069) | 0.010677 / 0.075646 (-0.064970) | 0.419970 / 0.419271 (0.000699) | 0.064626 / 0.043533 (0.021093) | 0.379536 / 0.255139 (0.124397) | 0.405790 / 0.283200 (0.122590) | 0.027290 / 0.141683 (-0.114393) | 1.884973 / 1.452155 (0.432819) | 1.960547 / 1.492716 (0.467831) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.259393 / 0.018006 (0.241386) | 0.502130 / 0.000490 (0.501640) | 0.013053 / 0.000200 (0.012853) | 0.000336 / 0.000054 (0.000281) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033459 / 0.037411 (-0.003953) | 0.135888 / 0.014526 (0.121362) | 0.145354 / 0.176557 (-0.031203) | 0.213289 / 0.737135 (-0.523847) | 0.151239 / 0.296338 (-0.145100) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.510817 / 0.215209 (0.295608) | 5.077888 / 2.077655 (3.000234) | 2.502991 / 1.504120 (0.998871) | 2.275566 / 1.541195 (0.734371) | 2.353025 / 1.468490 (0.884535) | 0.659062 / 4.584777 (-3.925715) | 4.411399 / 3.745712 (0.665686) | 2.227395 / 5.269862 (-3.042467) | 1.306771 / 4.565676 (-3.258905) | 0.081121 / 0.424275 (-0.343154) | 0.014252 / 0.007607 (0.006645) | 0.635040 / 0.226044 (0.408996) | 6.357500 / 2.268929 (4.088572) | 3.056647 / 55.444624 (-52.387977) | 2.671997 / 6.876477 (-4.204480) | 2.847955 / 2.142072 (0.705883) | 0.808163 / 4.805227 (-3.997064) | 0.177176 / 6.500664 (-6.323488) | 0.079984 / 0.075469 (0.004515) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.490471 / 1.841788 (-0.351317) | 17.927433 / 8.074308 (9.853124) | 17.744967 / 10.191392 (7.553575) | 0.171034 / 0.680424 (-0.509390) | 0.021432 / 0.534201 (-0.512769) | 0.515745 / 0.579283 (-0.063538) | 0.504746 / 0.434364 (0.070382) | 0.630862 / 0.540337 (0.090524) | 0.755275 / 1.386936 (-0.631662) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008227 / 0.011353 (-0.003126) | 0.004864 / 0.011008 (-0.006144) | 0.092801 / 0.038508 (0.054293) | 0.054996 / 0.023109 (0.031887) | 0.500348 / 0.275898 (0.224450) | 0.565028 / 0.323480 (0.241548) | 0.004792 / 0.007986 (-0.003194) | 0.005052 / 0.004328 (0.000723) | 0.090640 / 0.004250 (0.086390) | 0.074427 / 0.037052 (0.037374) | 0.499908 / 0.258489 (0.241419) | 0.566260 / 0.293841 (0.272419) | 0.040011 / 0.128546 (-0.088536) | 0.010438 / 0.075646 (-0.065208) | 0.099385 / 0.419271 (-0.319887) | 0.060485 / 0.043533 (0.016952) | 0.480603 / 0.255139 (0.225464) | 0.508807 / 0.283200 (0.225607) | 0.025976 / 0.141683 (-0.115707) | 1.870860 / 1.452155 (0.418705) | 1.943460 / 1.492716 (0.450744) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.227753 / 0.018006 (0.209747) | 0.501859 / 0.000490 (0.501369) | 0.008211 / 0.000200 (0.008011) | 0.000127 / 0.000054 (0.000073) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.038329 / 0.037411 (0.000918) | 0.148214 / 0.014526 (0.133688) | 0.162704 / 0.176557 (-0.013852) | 0.218543 / 0.737135 (-0.518592) | 0.162992 / 0.296338 (-0.133347) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.553195 / 0.215209 (0.337986) | 5.568080 / 2.077655 (3.490425) | 2.936616 / 1.504120 (1.432496) | 2.712624 / 1.541195 (1.171429) | 2.713245 / 1.468490 (1.244755) | 0.648593 / 4.584777 (-3.936184) | 4.641361 / 3.745712 (0.895648) | 2.207064 / 5.269862 (-3.062798) | 1.315325 / 4.565676 (-3.250351) | 0.080285 / 0.424275 (-0.343990) | 0.014143 / 0.007607 (0.006536) | 0.672467 / 0.226044 (0.446423) | 6.730262 / 2.268929 (4.461333) | 3.344468 / 55.444624 (-52.100157) | 2.927837 / 6.876477 (-3.948640) | 3.124735 / 2.142072 (0.982662) | 0.795894 / 4.805227 (-4.009333) | 0.170985 / 6.500664 (-6.329679) | 0.077406 / 0.075469 (0.001937) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.598059 / 1.841788 (-0.243729) | 18.531854 / 8.074308 (10.457546) | 18.394895 / 10.191392 (8.203503) | 0.195702 / 0.680424 (-0.484722) | 0.023633 / 0.534201 (-0.510568) | 0.518110 / 0.579283 (-0.061173) | 0.517773 / 0.434364 (0.083409) | 0.617902 / 0.540337 (0.077565) | 0.736459 / 1.386936 (-0.650477) |

</details>
</details>

![](https://cml.dev/watermark.png#5d4bb03237b74c0009043d50c5b4e4339cb98b2b "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006943 / 0.011353 (-0.004410) | 0.004524 / 0.011008 (-0.006485) | 0.121603 / 0.038508 (0.083095) | 0.047462 / 0.023109 (0.024353) | 0.362393 / 0.275898 (0.086495) | 0.440577 / 0.323480 (0.117098) | 0.004153 / 0.007986 (-0.003832) | 0.003778 / 0.004328 (-0.000550) | 0.090402 / 0.004250 (0.086152) | 0.066268 / 0.037052 (0.029216) | 0.380721 / 0.258489 (0.122232) | 0.442959 / 0.293841 (0.149118) | 0.035228 / 0.128546 (-0.093318) | 0.010217 / 0.075646 (-0.065429) | 0.408587 / 0.419271 (-0.010684) | 0.062609 / 0.043533 (0.019076) | 0.372682 / 0.255139 (0.117543) | 0.389270 / 0.283200 (0.106070) | 0.026699 / 0.141683 (-0.114984) | 1.760476 / 1.452155 (0.308321) | 1.795081 / 1.492716 (0.302365) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.229912 / 0.018006 (0.211906) | 0.476837 / 0.000490 (0.476348) | 0.008178 / 0.000200 (0.007978) | 0.000100 / 0.000054 (0.000045) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031116 / 0.037411 (-0.006296) | 0.126767 / 0.014526 (0.112241) | 0.134242 / 0.176557 (-0.042315) | 0.202120 / 0.737135 (-0.535016) | 0.142777 / 0.296338 (-0.153561) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.470690 / 0.215209 (0.255481) | 4.723198 / 2.077655 (2.645543) | 2.163870 / 1.504120 (0.659750) | 1.914177 / 1.541195 (0.372982) | 2.034529 / 1.468490 (0.566038) | 0.620472 / 4.584777 (-3.964305) | 4.391008 / 3.745712 (0.645296) | 2.100966 / 5.269862 (-3.168896) | 1.225945 / 4.565676 (-3.339732) | 0.076279 / 0.424275 (-0.347996) | 0.013551 / 0.007607 (0.005944) | 0.600989 / 0.226044 (0.374945) | 5.946715 / 2.268929 (3.677787) | 2.665117 / 55.444624 (-52.779508) | 2.320004 / 6.876477 (-4.556473) | 2.413131 / 2.142072 (0.271059) | 0.771908 / 4.805227 (-4.033320) | 0.165438 / 6.500664 (-6.335226) | 0.074512 / 0.075469 (-0.000957) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.432728 / 1.841788 (-0.409060) | 17.398133 / 8.074308 (9.323824) | 16.819152 / 10.191392 (6.627760) | 0.191849 / 0.680424 (-0.488575) | 0.021557 / 0.534201 (-0.512644) | 0.514380 / 0.579283 (-0.064903) | 0.501453 / 0.434364 (0.067089) | 0.634091 / 0.540337 (0.093753) | 0.756786 / 1.386936 (-0.630150) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007946 / 0.011353 (-0.003407) | 0.004751 / 0.011008 (-0.006257) | 0.090190 / 0.038508 (0.051682) | 0.052841 / 0.023109 (0.029732) | 0.480150 / 0.275898 (0.204252) | 0.537509 / 0.323480 (0.214029) | 0.004833 / 0.007986 (-0.003153) | 0.004796 / 0.004328 (0.000467) | 0.090616 / 0.004250 (0.086366) | 0.074325 / 0.037052 (0.037273) | 0.483776 / 0.258489 (0.225287) | 0.552094 / 0.293841 (0.258254) | 0.039240 / 0.128546 (-0.089307) | 0.010416 / 0.075646 (-0.065230) | 0.100275 / 0.419271 (-0.318996) | 0.058086 / 0.043533 (0.014553) | 0.468989 / 0.255139 (0.213850) | 0.485502 / 0.283200 (0.202302) | 0.027514 / 0.141683 (-0.114169) | 1.849625 / 1.452155 (0.397470) | 1.919515 / 1.492716 (0.426798) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.248061 / 0.018006 (0.230055) | 0.475630 / 0.000490 (0.475141) | 0.006248 / 0.000200 (0.006048) | 0.000105 / 0.000054 (0.000050) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.037746 / 0.037411 (0.000335) | 0.141638 / 0.014526 (0.127112) | 0.149530 / 0.176557 (-0.027026) | 0.209255 / 0.737135 (-0.527880) | 0.156447 / 0.296338 (-0.139892) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.544640 / 0.215209 (0.329431) | 5.493152 / 2.077655 (3.415497) | 2.869733 / 1.504120 (1.365613) | 2.624216 / 1.541195 (1.083022) | 2.710818 / 1.468490 (1.242328) | 0.640626 / 4.584777 (-3.944151) | 4.516130 / 3.745712 (0.770418) | 2.128097 / 5.269862 (-3.141765) | 1.278990 / 4.565676 (-3.286686) | 0.077114 / 0.424275 (-0.347161) | 0.013280 / 0.007607 (0.005673) | 0.655552 / 0.226044 (0.429507) | 6.526875 / 2.268929 (4.257947) | 3.347072 / 55.444624 (-52.097553) | 2.992435 / 6.876477 (-3.884041) | 3.124351 / 2.142072 (0.982278) | 0.778523 / 4.805227 (-4.026704) | 0.161873 / 6.500664 (-6.338791) | 0.072897 / 0.075469 (-0.002572) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.587058 / 1.841788 (-0.254730) | 18.170612 / 8.074308 (10.096304) | 17.220483 / 10.191392 (7.029091) | 0.207863 / 0.680424 (-0.472561) | 0.023746 / 0.534201 (-0.510455) | 0.512607 / 0.579283 (-0.066676) | 0.513258 / 0.434364 (0.078894) | 0.597880 / 0.540337 (0.057543) | 0.714974 / 1.386936 (-0.671962) |

</details>
</details>

![](https://cml.dev/watermark.png#98b1bdd492df953ca7139bb8c9a1771d5c603797 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006224 / 0.011353 (-0.005128) | 0.003857 / 0.011008 (-0.007151) | 0.099786 / 0.038508 (0.061278) | 0.037919 / 0.023109 (0.014810) | 0.315294 / 0.275898 (0.039396) | 0.390178 / 0.323480 (0.066698) | 0.005358 / 0.007986 (-0.002628) | 0.002989 / 0.004328 (-0.001340) | 0.077834 / 0.004250 (0.073583) | 0.053315 / 0.037052 (0.016263) | 0.325155 / 0.258489 (0.066666) | 0.374712 / 0.293841 (0.080871) | 0.029176 / 0.128546 (-0.099370) | 0.008658 / 0.075646 (-0.066988) | 0.314245 / 0.419271 (-0.105027) | 0.046684 / 0.043533 (0.003151) | 0.316473 / 0.255139 (0.061334) | 0.346119 / 0.283200 (0.062919) | 0.022452 / 0.141683 (-0.119230) | 1.540497 / 1.452155 (0.088343) | 1.594888 / 1.492716 (0.102172) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.204349 / 0.018006 (0.186343) | 0.426842 / 0.000490 (0.426353) | 0.003060 / 0.000200 (0.002860) | 0.000073 / 0.000054 (0.000019) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023611 / 0.037411 (-0.013801) | 0.100247 / 0.014526 (0.085721) | 0.107824 / 0.176557 (-0.068733) | 0.166845 / 0.737135 (-0.570291) | 0.112782 / 0.296338 (-0.183556) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.423053 / 0.215209 (0.207844) | 4.235553 / 2.077655 (2.157899) | 1.936589 / 1.504120 (0.432469) | 1.738519 / 1.541195 (0.197325) | 1.787905 / 1.468490 (0.319415) | 0.573362 / 4.584777 (-4.011414) | 3.395272 / 3.745712 (-0.350440) | 1.765977 / 5.269862 (-3.503884) | 1.049596 / 4.565676 (-3.516081) | 0.068868 / 0.424275 (-0.355407) | 0.011028 / 0.007607 (0.003421) | 0.532835 / 0.226044 (0.306791) | 5.314890 / 2.268929 (3.045962) | 2.368733 / 55.444624 (-53.075891) | 2.033959 / 6.876477 (-4.842518) | 2.130481 / 2.142072 (-0.011591) | 0.689360 / 4.805227 (-4.115867) | 0.140271 / 6.500664 (-6.360393) | 0.068198 / 0.075469 (-0.007271) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.237212 / 1.841788 (-0.604576) | 14.182215 / 8.074308 (6.107907) | 14.972608 / 10.191392 (4.781216) | 0.133977 / 0.680424 (-0.546447) | 0.016759 / 0.534201 (-0.517442) | 0.361552 / 0.579283 (-0.217731) | 0.394932 / 0.434364 (-0.039432) | 0.442601 / 0.540337 (-0.097736) | 0.535709 / 1.386936 (-0.851227) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006327 / 0.011353 (-0.005026) | 0.003780 / 0.011008 (-0.007228) | 0.078358 / 0.038508 (0.039850) | 0.037271 / 0.023109 (0.014162) | 0.456766 / 0.275898 (0.180868) | 0.515721 / 0.323480 (0.192241) | 0.004770 / 0.007986 (-0.003216) | 0.002942 / 0.004328 (-0.001387) | 0.077383 / 0.004250 (0.073132) | 0.051773 / 0.037052 (0.014721) | 0.460722 / 0.258489 (0.202233) | 0.519997 / 0.293841 (0.226157) | 0.030461 / 0.128546 (-0.098085) | 0.008622 / 0.075646 (-0.067024) | 0.083271 / 0.419271 (-0.336000) | 0.042242 / 0.043533 (-0.001291) | 0.447691 / 0.255139 (0.192552) | 0.481965 / 0.283200 (0.198765) | 0.019510 / 0.141683 (-0.122173) | 1.536718 / 1.452155 (0.084563) | 1.588433 / 1.492716 (0.095717) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.215880 / 0.018006 (0.197874) | 0.426102 / 0.000490 (0.425612) | 0.003976 / 0.000200 (0.003776) | 0.000079 / 0.000054 (0.000025) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026168 / 0.037411 (-0.011243) | 0.105786 / 0.014526 (0.091260) | 0.113772 / 0.176557 (-0.062785) | 0.166576 / 0.737135 (-0.570559) | 0.117560 / 0.296338 (-0.178779) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.490485 / 0.215209 (0.275276) | 4.890105 / 2.077655 (2.812450) | 2.515099 / 1.504120 (1.010979) | 2.306591 / 1.541195 (0.765396) | 2.383634 / 1.468490 (0.915144) | 0.573780 / 4.584777 (-4.010997) | 3.474394 / 3.745712 (-0.271318) | 1.746795 / 5.269862 (-3.523067) | 1.044678 / 4.565676 (-3.520998) | 0.069176 / 0.424275 (-0.355099) | 0.011045 / 0.007607 (0.003438) | 0.597234 / 0.226044 (0.371189) | 5.979614 / 2.268929 (3.710685) | 3.024203 / 55.444624 (-52.420422) | 2.687502 / 6.876477 (-4.188975) | 2.781637 / 2.142072 (0.639565) | 0.690482 / 4.805227 (-4.114745) | 0.150138 / 6.500664 (-6.350526) | 0.077076 / 0.075469 (0.001607) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.307501 / 1.841788 (-0.534287) | 14.366780 / 8.074308 (6.292471) | 14.966981 / 10.191392 (4.775589) | 0.153829 / 0.680424 (-0.526594) | 0.018047 / 0.534201 (-0.516154) | 0.361391 / 0.579283 (-0.217892) | 0.398345 / 0.434364 (-0.036019) | 0.424574 / 0.540337 (-0.115764) | 0.517165 / 1.386936 (-0.869771) |

</details>
</details>

![](https://cml.dev/watermark.png#98b1bdd492df953ca7139bb8c9a1771d5c603797 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006944 / 0.011353 (-0.004409) | 0.004504 / 0.011008 (-0.006504) | 0.105224 / 0.038508 (0.066716) | 0.047830 / 0.023109 (0.024721) | 0.339723 / 0.275898 (0.063825) | 0.419249 / 0.323480 (0.095769) | 0.005510 / 0.007986 (-0.002476) | 0.003574 / 0.004328 (-0.000754) | 0.079879 / 0.004250 (0.075628) | 0.066610 / 0.037052 (0.029557) | 0.353818 / 0.258489 (0.095329) | 0.397992 / 0.293841 (0.104151) | 0.031551 / 0.128546 (-0.096995) | 0.009037 / 0.075646 (-0.066610) | 0.355310 / 0.419271 (-0.063961) | 0.054931 / 0.043533 (0.011398) | 0.335153 / 0.255139 (0.080014) | 0.357460 / 0.283200 (0.074260) | 0.026031 / 0.141683 (-0.115652) | 1.546705 / 1.452155 (0.094550) | 1.627324 / 1.492716 (0.134608) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.276708 / 0.018006 (0.258701) | 0.589402 / 0.000490 (0.588912) | 0.009560 / 0.000200 (0.009360) | 0.000095 / 0.000054 (0.000041) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031041 / 0.037411 (-0.006370) | 0.117219 / 0.014526 (0.102693) | 0.125200 / 0.176557 (-0.051356) | 0.181528 / 0.737135 (-0.555607) | 0.131898 / 0.296338 (-0.164440) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.409965 / 0.215209 (0.194756) | 4.102700 / 2.077655 (2.025045) | 1.887578 / 1.504120 (0.383458) | 1.696490 / 1.541195 (0.155295) | 1.821352 / 1.468490 (0.352862) | 0.545422 / 4.584777 (-4.039355) | 3.933784 / 3.745712 (0.188071) | 1.934254 / 5.269862 (-3.335607) | 1.114935 / 4.565676 (-3.450742) | 0.067615 / 0.424275 (-0.356660) | 0.012004 / 0.007607 (0.004397) | 0.522048 / 0.226044 (0.296004) | 5.209224 / 2.268929 (2.940296) | 2.369911 / 55.444624 (-53.074714) | 2.032960 / 6.876477 (-4.843517) | 2.228874 / 2.142072 (0.086802) | 0.673172 / 4.805227 (-4.132055) | 0.147017 / 6.500664 (-6.353647) | 0.067020 / 0.075469 (-0.008449) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.281490 / 1.841788 (-0.560298) | 16.129701 / 8.074308 (8.055393) | 15.474730 / 10.191392 (5.283338) | 0.143934 / 0.680424 (-0.536490) | 0.018311 / 0.534201 (-0.515890) | 0.435940 / 0.579283 (-0.143343) | 0.446846 / 0.434364 (0.012482) | 0.543943 / 0.540337 (0.003605) | 0.648041 / 1.386936 (-0.738895) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007380 / 0.011353 (-0.003973) | 0.004510 / 0.011008 (-0.006499) | 0.080741 / 0.038508 (0.042233) | 0.050907 / 0.023109 (0.027797) | 0.425548 / 0.275898 (0.149650) | 0.487959 / 0.323480 (0.164479) | 0.005887 / 0.007986 (-0.002099) | 0.003689 / 0.004328 (-0.000639) | 0.079588 / 0.004250 (0.075338) | 0.071841 / 0.037052 (0.034788) | 0.425172 / 0.258489 (0.166683) | 0.471185 / 0.293841 (0.177344) | 0.035768 / 0.128546 (-0.092779) | 0.009229 / 0.075646 (-0.066418) | 0.086021 / 0.419271 (-0.333250) | 0.052424 / 0.043533 (0.008891) | 0.413634 / 0.255139 (0.158495) | 0.422310 / 0.283200 (0.139111) | 0.026019 / 0.141683 (-0.115664) | 1.616861 / 1.452155 (0.164707) | 1.653660 / 1.492716 (0.160943) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.280096 / 0.018006 (0.262090) | 0.587853 / 0.000490 (0.587363) | 0.006560 / 0.000200 (0.006360) | 0.000181 / 0.000054 (0.000127) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033747 / 0.037411 (-0.003665) | 0.125089 / 0.014526 (0.110564) | 0.137995 / 0.176557 (-0.038561) | 0.188192 / 0.737135 (-0.548943) | 0.141438 / 0.296338 (-0.154900) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.471524 / 0.215209 (0.256315) | 4.713988 / 2.077655 (2.636334) | 2.414785 / 1.504120 (0.910665) | 2.226815 / 1.541195 (0.685620) | 2.259222 / 1.468490 (0.790732) | 0.551663 / 4.584777 (-4.033114) | 4.031399 / 3.745712 (0.285686) | 1.966917 / 5.269862 (-3.302945) | 1.154487 / 4.565676 (-3.411190) | 0.068500 / 0.424275 (-0.355775) | 0.012127 / 0.007607 (0.004520) | 0.579342 / 0.226044 (0.353298) | 5.757415 / 2.268929 (3.488486) | 2.820012 / 55.444624 (-52.624613) | 2.521783 / 6.876477 (-4.354694) | 2.699994 / 2.142072 (0.557921) | 0.686152 / 4.805227 (-4.119075) | 0.148521 / 6.500664 (-6.352143) | 0.068478 / 0.075469 (-0.006991) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.336260 / 1.841788 (-0.505528) | 17.016935 / 8.074308 (8.942627) | 16.406951 / 10.191392 (6.215559) | 0.166907 / 0.680424 (-0.513517) | 0.020166 / 0.534201 (-0.514035) | 0.437690 / 0.579283 (-0.141593) | 0.480337 / 0.434364 (0.045973) | 0.518065 / 0.540337 (-0.022272) | 0.625904 / 1.386936 (-0.761032) |

</details>
</details>

![](https://cml.dev/watermark.png#98b1bdd492df953ca7139bb8c9a1771d5c603797 "CML watermark")