Add writer_batch_size for ArrowBasedBuilder
Opened by lhoestq (closed)

This way we can control the size of the record_batches/row_groups of arrow/parquet files.

This can be useful for `datasets-server` to keep control of the row groups size which can affect random access performance for audio/image/video datasets

Right now having 1,000 examples per row group cause some image datasets to be pretty slow for random access (e.g. 4seconds for `beans` to get 20 rows)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008745 / 0.011353 (-0.002608) | 0.004651 / 0.011008 (-0.006357) | 0.099678 / 0.038508 (0.061170) | 0.029441 / 0.023109 (0.006332) | 0.300314 / 0.275898 (0.024416) | 0.342022 / 0.323480 (0.018542) | 0.006965 / 0.007986 (-0.001021) | 0.003382 / 0.004328 (-0.000946) | 0.078195 / 0.004250 (0.073945) | 0.033308 / 0.037052 (-0.003744) | 0.300857 / 0.258489 (0.042368) | 0.356763 / 0.293841 (0.062922) | 0.033919 / 0.128546 (-0.094627) | 0.011436 / 0.075646 (-0.064210) | 0.319581 / 0.419271 (-0.099691) | 0.041303 / 0.043533 (-0.002229) | 0.299387 / 0.255139 (0.044248) | 0.327783 / 0.283200 (0.044583) | 0.087210 / 0.141683 (-0.054473) | 1.498757 / 1.452155 (0.046603) | 1.560417 / 1.492716 (0.067701) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.191806 / 0.018006 (0.173800) | 0.407044 / 0.000490 (0.406554) | 0.005116 / 0.000200 (0.004916) | 0.000073 / 0.000054 (0.000019) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023760 / 0.037411 (-0.013652) | 0.096844 / 0.014526 (0.082318) | 0.104710 / 0.176557 (-0.071847) | 0.168161 / 0.737135 (-0.568974) | 0.107808 / 0.296338 (-0.188531) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.417707 / 0.215209 (0.202498) | 4.155952 / 2.077655 (2.078297) | 1.864934 / 1.504120 (0.360814) | 1.654925 / 1.541195 (0.113730) | 1.731341 / 1.468490 (0.262851) | 0.692014 / 4.584777 (-3.892763) | 3.407318 / 3.745712 (-0.338394) | 3.394791 / 5.269862 (-1.875071) | 1.650429 / 4.565676 (-2.915247) | 0.082177 / 0.424275 (-0.342098) | 0.012463 / 0.007607 (0.004856) | 0.523681 / 0.226044 (0.297637) | 5.249426 / 2.268929 (2.980498) | 2.327443 / 55.444624 (-53.117181) | 1.982160 / 6.876477 (-4.894317) | 2.019822 / 2.142072 (-0.122250) | 0.804820 / 4.805227 (-4.000408) | 0.148423 / 6.500664 (-6.352241) | 0.064938 / 0.075469 (-0.010531) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.225722 / 1.841788 (-0.616066) | 13.774257 / 8.074308 (5.699949) | 14.090298 / 10.191392 (3.898906) | 0.152489 / 0.680424 (-0.527935) | 0.028595 / 0.534201 (-0.505606) | 0.399011 / 0.579283 (-0.180272) | 0.399546 / 0.434364 (-0.034818) | 0.485513 / 0.540337 (-0.054824) | 0.564055 / 1.386936 (-0.822881) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006891 / 0.011353 (-0.004462) | 0.004557 / 0.011008 (-0.006451) | 0.077868 / 0.038508 (0.039360) | 0.028767 / 0.023109 (0.005657) | 0.344127 / 0.275898 (0.068229) | 0.377097 / 0.323480 (0.053617) | 0.005119 / 0.007986 (-0.002866) | 0.003547 / 0.004328 (-0.000782) | 0.077047 / 0.004250 (0.072796) | 0.043037 / 0.037052 (0.005984) | 0.341900 / 0.258489 (0.083410) | 0.384570 / 0.293841 (0.090729) | 0.032606 / 0.128546 (-0.095940) | 0.011752 / 0.075646 (-0.063894) | 0.086731 / 0.419271 (-0.332540) | 0.045459 / 0.043533 (0.001926) | 0.339308 / 0.255139 (0.084169) | 0.370498 / 0.283200 (0.087298) | 0.096237 / 0.141683 (-0.045446) | 1.499253 / 1.452155 (0.047098) | 1.583871 / 1.492716 (0.091154) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.245471 / 0.018006 (0.227465) | 0.408750 / 0.000490 (0.408260) | 0.008992 / 0.000200 (0.008792) | 0.000249 / 0.000054 (0.000194) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025508 / 0.037411 (-0.011903) | 0.102103 / 0.014526 (0.087578) | 0.109247 / 0.176557 (-0.067310) | 0.176369 / 0.737135 (-0.560766) | 0.111241 / 0.296338 (-0.185097) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.437209 / 0.215209 (0.222000) | 4.354386 / 2.077655 (2.276731) | 2.064008 / 1.504120 (0.559888) | 1.855518 / 1.541195 (0.314323) | 1.931647 / 1.468490 (0.463157) | 0.704913 / 4.584777 (-3.879864) | 3.397913 / 3.745712 (-0.347800) | 1.871524 / 5.269862 (-3.398338) | 1.176492 / 4.565676 (-3.389185) | 0.083976 / 0.424275 (-0.340299) | 0.012806 / 0.007607 (0.005199) | 0.539138 / 0.226044 (0.313094) | 5.401493 / 2.268929 (3.132564) | 2.539185 / 55.444624 (-52.905440) | 2.186445 / 6.876477 (-4.690031) | 2.222170 / 2.142072 (0.080097) | 0.815641 / 4.805227 (-3.989586) | 0.153033 / 6.500664 (-6.347631) | 0.069168 / 0.075469 (-0.006301) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.283530 / 1.841788 (-0.558258) | 14.075831 / 8.074308 (6.001523) | 13.649137 / 10.191392 (3.457745) | 0.127517 / 0.680424 (-0.552907) | 0.016619 / 0.534201 (-0.517582) | 0.377400 / 0.579283 (-0.201883) | 0.410796 / 0.434364 (-0.023568) | 0.463996 / 0.540337 (-0.076342) | 0.551867 / 1.386936 (-0.835069) |

</details>
</details>

![](https://cml.dev/watermark.png#1135285d80ff9cd65fc51905f08343b4d7c2fa9c "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009161 / 0.011353 (-0.002192) | 0.004987 / 0.011008 (-0.006022) | 0.098553 / 0.038508 (0.060045) | 0.034326 / 0.023109 (0.011216) | 0.295325 / 0.275898 (0.019427) | 0.326361 / 0.323480 (0.002881) | 0.007827 / 0.007986 (-0.000159) | 0.004933 / 0.004328 (0.000604) | 0.074236 / 0.004250 (0.069986) | 0.040410 / 0.037052 (0.003357) | 0.295644 / 0.258489 (0.037155) | 0.355050 / 0.293841 (0.061209) | 0.038384 / 0.128546 (-0.090162) | 0.011845 / 0.075646 (-0.063801) | 0.340678 / 0.419271 (-0.078594) | 0.047615 / 0.043533 (0.004082) | 0.292429 / 0.255139 (0.037290) | 0.312610 / 0.283200 (0.029410) | 0.100106 / 0.141683 (-0.041577) | 1.446186 / 1.452155 (-0.005969) | 1.534763 / 1.492716 (0.042046) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.213667 / 0.018006 (0.195661) | 0.447310 / 0.000490 (0.446820) | 0.000402 / 0.000200 (0.000202) | 0.000056 / 0.000054 (0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027604 / 0.037411 (-0.009807) | 0.112785 / 0.014526 (0.098259) | 0.119450 / 0.176557 (-0.057106) | 0.185728 / 0.737135 (-0.551407) | 0.122860 / 0.296338 (-0.173478) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.399162 / 0.215209 (0.183953) | 3.992701 / 2.077655 (1.915046) | 1.773881 / 1.504120 (0.269761) | 1.589842 / 1.541195 (0.048647) | 1.670065 / 1.468490 (0.201575) | 0.707669 / 4.584777 (-3.877107) | 3.719657 / 3.745712 (-0.026055) | 2.139629 / 5.269862 (-3.130232) | 1.467224 / 4.565676 (-3.098453) | 0.086033 / 0.424275 (-0.338242) | 0.012151 / 0.007607 (0.004544) | 0.519700 / 0.226044 (0.293656) | 5.150254 / 2.268929 (2.881325) | 2.305076 / 55.444624 (-53.139548) | 1.927914 / 6.876477 (-4.948563) | 1.999461 / 2.142072 (-0.142612) | 0.851819 / 4.805227 (-3.953408) | 0.165513 / 6.500664 (-6.335151) | 0.061898 / 0.075469 (-0.013571) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.226251 / 1.841788 (-0.615536) | 14.990253 / 8.074308 (6.915945) | 14.658720 / 10.191392 (4.467328) | 0.191665 / 0.680424 (-0.488759) | 0.028768 / 0.534201 (-0.505433) | 0.443907 / 0.579283 (-0.135376) | 0.455183 / 0.434364 (0.020819) | 0.552760 / 0.540337 (0.012422) | 0.653927 / 1.386936 (-0.733009) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007677 / 0.011353 (-0.003675) | 0.005340 / 0.011008 (-0.005668) | 0.075644 / 0.038508 (0.037136) | 0.035046 / 0.023109 (0.011937) | 0.341437 / 0.275898 (0.065538) | 0.377782 / 0.323480 (0.054302) | 0.006091 / 0.007986 (-0.001895) | 0.004170 / 0.004328 (-0.000158) | 0.074294 / 0.004250 (0.070044) | 0.049851 / 0.037052 (0.012798) | 0.351691 / 0.258489 (0.093202) | 0.386020 / 0.293841 (0.092179) | 0.036884 / 0.128546 (-0.091662) | 0.012475 / 0.075646 (-0.063172) | 0.087267 / 0.419271 (-0.332005) | 0.058623 / 0.043533 (0.015090) | 0.347186 / 0.255139 (0.092047) | 0.355869 / 0.283200 (0.072669) | 0.112022 / 0.141683 (-0.029661) | 1.451798 / 1.452155 (-0.000357) | 1.553262 / 1.492716 (0.060546) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.233451 / 0.018006 (0.215445) | 0.444384 / 0.000490 (0.443895) | 0.003695 / 0.000200 (0.003495) | 0.000088 / 0.000054 (0.000034) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029686 / 0.037411 (-0.007725) | 0.113736 / 0.014526 (0.099210) | 0.123998 / 0.176557 (-0.052559) | 0.197847 / 0.737135 (-0.539288) | 0.129936 / 0.296338 (-0.166403) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.421904 / 0.215209 (0.206695) | 4.203533 / 2.077655 (2.125878) | 2.038199 / 1.504120 (0.534079) | 1.832402 / 1.541195 (0.291208) | 1.930765 / 1.468490 (0.462274) | 0.709775 / 4.584777 (-3.875002) | 3.760893 / 3.745712 (0.015181) | 2.091185 / 5.269862 (-3.178677) | 1.342248 / 4.565676 (-3.223428) | 0.087770 / 0.424275 (-0.336505) | 0.012357 / 0.007607 (0.004750) | 0.519605 / 0.226044 (0.293560) | 5.215883 / 2.268929 (2.946954) | 2.510200 / 55.444624 (-52.934425) | 2.192482 / 6.876477 (-4.683995) | 2.290214 / 2.142072 (0.148141) | 0.872067 / 4.805227 (-3.933160) | 0.168491 / 6.500664 (-6.332173) | 0.064707 / 0.075469 (-0.010762) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.291956 / 1.841788 (-0.549832) | 15.244530 / 8.074308 (7.170222) | 13.594895 / 10.191392 (3.403503) | 0.172669 / 0.680424 (-0.507755) | 0.017765 / 0.534201 (-0.516436) | 0.426946 / 0.579283 (-0.152337) | 0.442843 / 0.434364 (0.008479) | 0.549683 / 0.540337 (0.009346) | 0.653433 / 1.386936 (-0.733503) |

</details>
</details>

![](https://cml.dev/watermark.png#b54a6d21795cf6cc50a13ff870648241a60fd2e0 "CML watermark")


Can you review this @mariosasko ? since Albert is off

<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008396 / 0.011353 (-0.002957) | 0.004556 / 0.011008 (-0.006452) | 0.101343 / 0.038508 (0.062835) | 0.029137 / 0.023109 (0.006027) | 0.298553 / 0.275898 (0.022655) | 0.334050 / 0.323480 (0.010570) | 0.006746 / 0.007986 (-0.001239) | 0.005050 / 0.004328 (0.000721) | 0.076055 / 0.004250 (0.071804) | 0.031988 / 0.037052 (-0.005064) | 0.301324 / 0.258489 (0.042835) | 0.340121 / 0.293841 (0.046280) | 0.033827 / 0.128546 (-0.094720) | 0.011447 / 0.075646 (-0.064200) | 0.321827 / 0.419271 (-0.097445) | 0.040846 / 0.043533 (-0.002687) | 0.296957 / 0.255139 (0.041818) | 0.324178 / 0.283200 (0.040979) | 0.083852 / 0.141683 (-0.057831) | 1.456123 / 1.452155 (0.003968) | 1.538311 / 1.492716 (0.045595) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.208897 / 0.018006 (0.190891) | 0.430560 / 0.000490 (0.430070) | 0.002917 / 0.000200 (0.002717) | 0.000077 / 0.000054 (0.000022) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024332 / 0.037411 (-0.013080) | 0.101659 / 0.014526 (0.087133) | 0.107636 / 0.176557 (-0.068920) | 0.168805 / 0.737135 (-0.568330) | 0.111404 / 0.296338 (-0.184934) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.412704 / 0.215209 (0.197495) | 4.124852 / 2.077655 (2.047197) | 1.843555 / 1.504120 (0.339435) | 1.641636 / 1.541195 (0.100441) | 1.755783 / 1.468490 (0.287293) | 0.693212 / 4.584777 (-3.891565) | 3.391803 / 3.745712 (-0.353909) | 1.954473 / 5.269862 (-3.315389) | 1.274395 / 4.565676 (-3.291282) | 0.082536 / 0.424275 (-0.341739) | 0.012335 / 0.007607 (0.004728) | 0.523720 / 0.226044 (0.297676) | 5.268339 / 2.268929 (2.999411) | 2.318163 / 55.444624 (-53.126461) | 1.978503 / 6.876477 (-4.897974) | 2.046689 / 2.142072 (-0.095384) | 0.806735 / 4.805227 (-3.998492) | 0.148010 / 6.500664 (-6.352654) | 0.065305 / 0.075469 (-0.010164) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.266950 / 1.841788 (-0.574838) | 13.870803 / 8.074308 (5.796495) | 14.272556 / 10.191392 (4.081164) | 0.151703 / 0.680424 (-0.528720) | 0.028991 / 0.534201 (-0.505210) | 0.400831 / 0.579283 (-0.178452) | 0.400891 / 0.434364 (-0.033473) | 0.476225 / 0.540337 (-0.064113) | 0.564925 / 1.386936 (-0.822011) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006810 / 0.011353 (-0.004543) | 0.004544 / 0.011008 (-0.006464) | 0.076516 / 0.038508 (0.038008) | 0.027705 / 0.023109 (0.004596) | 0.343215 / 0.275898 (0.067317) | 0.379136 / 0.323480 (0.055656) | 0.005227 / 0.007986 (-0.002758) | 0.003527 / 0.004328 (-0.000801) | 0.074775 / 0.004250 (0.070524) | 0.041700 / 0.037052 (0.004648) | 0.343612 / 0.258489 (0.085123) | 0.385657 / 0.293841 (0.091817) | 0.032082 / 0.128546 (-0.096464) | 0.011567 / 0.075646 (-0.064079) | 0.083814 / 0.419271 (-0.335458) | 0.042173 / 0.043533 (-0.001360) | 0.340261 / 0.255139 (0.085122) | 0.364778 / 0.283200 (0.081578) | 0.093401 / 0.141683 (-0.048282) | 1.513475 / 1.452155 (0.061320) | 1.599393 / 1.492716 (0.106677) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.237117 / 0.018006 (0.219111) | 0.424241 / 0.000490 (0.423751) | 0.002900 / 0.000200 (0.002700) | 0.000076 / 0.000054 (0.000021) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031122 / 0.037411 (-0.006289) | 0.107530 / 0.014526 (0.093004) | 0.117777 / 0.176557 (-0.058780) | 0.188300 / 0.737135 (-0.548836) | 0.119989 / 0.296338 (-0.176349) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.438563 / 0.215209 (0.223354) | 4.404969 / 2.077655 (2.327315) | 2.260182 / 1.504120 (0.756062) | 2.035472 / 1.541195 (0.494277) | 2.045685 / 1.468490 (0.577195) | 0.706758 / 4.584777 (-3.878019) | 3.434843 / 3.745712 (-0.310869) | 1.909533 / 5.269862 (-3.360328) | 1.175374 / 4.565676 (-3.390303) | 0.084831 / 0.424275 (-0.339444) | 0.012441 / 0.007607 (0.004833) | 0.551818 / 0.226044 (0.325774) | 5.509005 / 2.268929 (3.240077) | 2.576545 / 55.444624 (-52.868080) | 2.226204 / 6.876477 (-4.650272) | 2.276544 / 2.142072 (0.134471) | 0.818069 / 4.805227 (-3.987158) | 0.152797 / 6.500664 (-6.347867) | 0.067896 / 0.075469 (-0.007573) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.276859 / 1.841788 (-0.564929) | 14.312914 / 8.074308 (6.238606) | 13.406602 / 10.191392 (3.215210) | 0.157466 / 0.680424 (-0.522958) | 0.016709 / 0.534201 (-0.517492) | 0.390951 / 0.579283 (-0.188333) | 0.395525 / 0.434364 (-0.038839) | 0.484486 / 0.540337 (-0.055852) | 0.576125 / 1.386936 (-0.810811) |

</details>
</details>

![](https://cml.dev/watermark.png#b951e1b6cdd927604599f1aa5dadfb8ee8e62e05 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007316 / 0.011353 (-0.004037) | 0.005041 / 0.011008 (-0.005968) | 0.100477 / 0.038508 (0.061969) | 0.034068 / 0.023109 (0.010959) | 0.351156 / 0.275898 (0.075258) | 0.373892 / 0.323480 (0.050412) | 0.005748 / 0.007986 (-0.002237) | 0.003959 / 0.004328 (-0.000370) | 0.075540 / 0.004250 (0.071290) | 0.045282 / 0.037052 (0.008230) | 0.362364 / 0.258489 (0.103874) | 0.376461 / 0.293841 (0.082620) | 0.036724 / 0.128546 (-0.091822) | 0.012008 / 0.075646 (-0.063638) | 0.333802 / 0.419271 (-0.085470) | 0.050107 / 0.043533 (0.006574) | 0.348003 / 0.255139 (0.092864) | 0.367187 / 0.283200 (0.083988) | 0.103171 / 0.141683 (-0.038511) | 1.448281 / 1.452155 (-0.003874) | 1.516231 / 1.492716 (0.023514) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.203651 / 0.018006 (0.185645) | 0.438103 / 0.000490 (0.437613) | 0.004165 / 0.000200 (0.003966) | 0.000085 / 0.000054 (0.000030) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027068 / 0.037411 (-0.010343) | 0.111728 / 0.014526 (0.097202) | 0.116963 / 0.176557 (-0.059594) | 0.172652 / 0.737135 (-0.564483) | 0.124257 / 0.296338 (-0.172082) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.407937 / 0.215209 (0.192728) | 4.066008 / 2.077655 (1.988353) | 1.895000 / 1.504120 (0.390880) | 1.698422 / 1.541195 (0.157227) | 1.872446 / 1.468490 (0.403956) | 0.688888 / 4.584777 (-3.895889) | 3.743635 / 3.745712 (-0.002077) | 2.161507 / 5.269862 (-3.108354) | 1.485218 / 4.565676 (-3.080458) | 0.085959 / 0.424275 (-0.338316) | 0.012554 / 0.007607 (0.004947) | 0.510953 / 0.226044 (0.284909) | 5.103241 / 2.268929 (2.834312) | 2.439670 / 55.444624 (-53.004955) | 2.057089 / 6.876477 (-4.819387) | 2.240137 / 2.142072 (0.098065) | 0.847750 / 4.805227 (-3.957477) | 0.172952 / 6.500664 (-6.327712) | 0.066023 / 0.075469 (-0.009446) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.190677 / 1.841788 (-0.651110) | 14.593162 / 8.074308 (6.518854) | 14.254983 / 10.191392 (4.063591) | 0.155811 / 0.680424 (-0.524613) | 0.017698 / 0.534201 (-0.516503) | 0.420455 / 0.579283 (-0.158828) | 0.412146 / 0.434364 (-0.022218) | 0.493113 / 0.540337 (-0.047225) | 0.582097 / 1.386936 (-0.804839) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007319 / 0.011353 (-0.004033) | 0.005102 / 0.011008 (-0.005906) | 0.073760 / 0.038508 (0.035252) | 0.033496 / 0.023109 (0.010387) | 0.338778 / 0.275898 (0.062880) | 0.371870 / 0.323480 (0.048391) | 0.005804 / 0.007986 (-0.002182) | 0.004142 / 0.004328 (-0.000186) | 0.073203 / 0.004250 (0.068953) | 0.046568 / 0.037052 (0.009516) | 0.343544 / 0.258489 (0.085055) | 0.381188 / 0.293841 (0.087347) | 0.036391 / 0.128546 (-0.092155) | 0.012046 / 0.075646 (-0.063600) | 0.086007 / 0.419271 (-0.333265) | 0.048706 / 0.043533 (0.005173) | 0.330836 / 0.255139 (0.075697) | 0.355328 / 0.283200 (0.072128) | 0.100104 / 0.141683 (-0.041579) | 1.434237 / 1.452155 (-0.017917) | 1.549380 / 1.492716 (0.056663) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.231099 / 0.018006 (0.213093) | 0.450650 / 0.000490 (0.450160) | 0.000404 / 0.000200 (0.000204) | 0.000059 / 0.000054 (0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030534 / 0.037411 (-0.006877) | 0.119005 / 0.014526 (0.104479) | 0.125362 / 0.176557 (-0.051195) | 0.176823 / 0.737135 (-0.560313) | 0.132044 / 0.296338 (-0.164295) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.431004 / 0.215209 (0.215795) | 4.318969 / 2.077655 (2.241315) | 1.994941 / 1.504120 (0.490821) | 1.791870 / 1.541195 (0.250675) | 1.904134 / 1.468490 (0.435644) | 0.723493 / 4.584777 (-3.861284) | 3.823670 / 3.745712 (0.077958) | 2.118892 / 5.269862 (-3.150969) | 1.375088 / 4.565676 (-3.190588) | 0.088875 / 0.424275 (-0.335400) | 0.013137 / 0.007607 (0.005530) | 0.530523 / 0.226044 (0.304479) | 5.341438 / 2.268929 (3.072509) | 2.459044 / 55.444624 (-52.985580) | 2.150119 / 6.876477 (-4.726357) | 2.228567 / 2.142072 (0.086494) | 0.877549 / 4.805227 (-3.927678) | 0.175040 / 6.500664 (-6.325625) | 0.068188 / 0.075469 (-0.007281) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.273780 / 1.841788 (-0.568008) | 15.206331 / 8.074308 (7.132023) | 14.963058 / 10.191392 (4.771666) | 0.184543 / 0.680424 (-0.495881) | 0.017612 / 0.534201 (-0.516589) | 0.426248 / 0.579283 (-0.153035) | 0.437889 / 0.434364 (0.003525) | 0.508979 / 0.540337 (-0.031359) | 0.602040 / 1.386936 (-0.784896) |

</details>
</details>

![](https://cml.dev/watermark.png#c5ca1d86949ec3a5fdaec03b80500fb822bcfab4 "CML watermark")