Use parquet export if possible
Opened by lhoestq (closed)

The idea is to make this code work for datasets with scripts if they have a Parquet export

```python
ds = load_dataset("squad", trust_remote_code=False)
```

And more generally, it means we use the Parquet export whenever it's possible (it's safer and faster than dataset scripts).

I also added a `config.USE_PARQUET_EXPORT` variable to use in the datasets-server parquet conversion job

- [x] Needs https://github.com/huggingface/datasets/pull/6429 to be merged first

cc @severo I use the /parquet and /info endpoints from datasets-server

Comments:
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005177 / 0.011353 (-0.006176) | 0.003002 / 0.011008 (-0.008006) | 0.061915 / 0.038508 (0.023407) | 0.052065 / 0.023109 (0.028956) | 0.246114 / 0.275898 (-0.029784) | 0.273974 / 0.323480 (-0.049506) | 0.002983 / 0.007986 (-0.005003) | 0.002444 / 0.004328 (-0.001885) | 0.048424 / 0.004250 (0.044174) | 0.039609 / 0.037052 (0.002557) | 0.257771 / 0.258489 (-0.000718) | 0.286228 / 0.293841 (-0.007613) | 0.023925 / 0.128546 (-0.104621) | 0.007248 / 0.075646 (-0.068398) | 0.202205 / 0.419271 (-0.217067) | 0.037124 / 0.043533 (-0.006409) | 0.254872 / 0.255139 (-0.000267) | 0.275252 / 0.283200 (-0.007947) | 0.019251 / 0.141683 (-0.122432) | 1.074921 / 1.452155 (-0.377234) | 1.146515 / 1.492716 (-0.346202) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091998 / 0.018006 (0.073992) | 0.299146 / 0.000490 (0.298656) | 0.000240 / 0.000200 (0.000040) | 0.000054 / 0.000054 (0.000000) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019266 / 0.037411 (-0.018145) | 0.062560 / 0.014526 (0.048034) | 0.075012 / 0.176557 (-0.101544) | 0.120077 / 0.737135 (-0.617058) | 0.077851 / 0.296338 (-0.218488) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.290629 / 0.215209 (0.075420) | 2.823847 / 2.077655 (0.746192) | 1.516966 / 1.504120 (0.012846) | 1.393383 / 1.541195 (-0.147812) | 1.427688 / 1.468490 (-0.040802) | 0.407456 / 4.584777 (-4.177321) | 2.378280 / 3.745712 (-1.367433) | 2.689800 / 5.269862 (-2.580061) | 1.588037 / 4.565676 (-2.977640) | 0.045837 / 0.424275 (-0.378438) | 0.004884 / 0.007607 (-0.002724) | 0.340464 / 0.226044 (0.114420) | 3.377158 / 2.268929 (1.108230) | 1.897854 / 55.444624 (-53.546771) | 1.588285 / 6.876477 (-5.288191) | 1.651708 / 2.142072 (-0.490364) | 0.482018 / 4.805227 (-4.323209) | 0.101583 / 6.500664 (-6.399081) | 0.042306 / 0.075469 (-0.033163) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.948659 / 1.841788 (-0.893128) | 11.809778 / 8.074308 (3.735470) | 10.481896 / 10.191392 (0.290504) | 0.143538 / 0.680424 (-0.536885) | 0.014105 / 0.534201 (-0.520096) | 0.272278 / 0.579283 (-0.307005) | 0.264241 / 0.434364 (-0.170123) | 0.307187 / 0.540337 (-0.233150) | 0.401270 / 1.386936 (-0.985666) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004831 / 0.011353 (-0.006521) | 0.002896 / 0.011008 (-0.008112) | 0.047479 / 0.038508 (0.008971) | 0.050665 / 0.023109 (0.027555) | 0.275243 / 0.275898 (-0.000655) | 0.296547 / 0.323480 (-0.026933) | 0.004022 / 0.007986 (-0.003963) | 0.002425 / 0.004328 (-0.001904) | 0.047086 / 0.004250 (0.042836) | 0.039611 / 0.037052 (0.002558) | 0.275272 / 0.258489 (0.016783) | 0.302429 / 0.293841 (0.008588) | 0.024308 / 0.128546 (-0.104238) | 0.007167 / 0.075646 (-0.068479) | 0.052825 / 0.419271 (-0.366446) | 0.032319 / 0.043533 (-0.011213) | 0.273334 / 0.255139 (0.018195) | 0.291161 / 0.283200 (0.007961) | 0.017918 / 0.141683 (-0.123764) | 1.110005 / 1.452155 (-0.342150) | 1.176616 / 1.492716 (-0.316100) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092478 / 0.018006 (0.074471) | 0.311431 / 0.000490 (0.310942) | 0.000237 / 0.000200 (0.000037) | 0.000059 / 0.000054 (0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021979 / 0.037411 (-0.015432) | 0.080617 / 0.014526 (0.066091) | 0.081534 / 0.176557 (-0.095023) | 0.121073 / 0.737135 (-0.616062) | 0.083235 / 0.296338 (-0.213104) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.289527 / 0.215209 (0.074318) | 2.839668 / 2.077655 (0.762013) | 1.601737 / 1.504120 (0.097617) | 1.496028 / 1.541195 (-0.045167) | 1.511933 / 1.468490 (0.043443) | 0.399819 / 4.584777 (-4.184958) | 2.394147 / 3.745712 (-1.351565) | 2.520767 / 5.269862 (-2.749095) | 1.589496 / 4.565676 (-2.976180) | 0.046673 / 0.424275 (-0.377602) | 0.004858 / 0.007607 (-0.002749) | 0.357986 / 0.226044 (0.131941) | 3.376217 / 2.268929 (1.107289) | 1.981853 / 55.444624 (-53.462771) | 1.682240 / 6.876477 (-5.194236) | 1.830643 / 2.142072 (-0.311429) | 0.478286 / 4.805227 (-4.326941) | 0.099589 / 6.500664 (-6.401075) | 0.041173 / 0.075469 (-0.034296) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.985160 / 1.841788 (-0.856628) | 12.312963 / 8.074308 (4.238655) | 10.577225 / 10.191392 (0.385833) | 0.130167 / 0.680424 (-0.550257) | 0.016657 / 0.534201 (-0.517544) | 0.271330 / 0.579283 (-0.307953) | 0.276979 / 0.434364 (-0.157385) | 0.304904 / 0.540337 (-0.235434) | 0.412090 / 1.386936 (-0.974846) |

</details>
</details>

![](https://cml.dev/watermark.png#1adc80151e892122ecb60f4e0b4572b136b2dd47 "CML watermark")


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6448). All of your documentation changes will be reflected on that endpoint.

hooray! very excited about this

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005039 / 0.011353 (-0.006314) | 0.003577 / 0.011008 (-0.007431) | 0.062892 / 0.038508 (0.024384) | 0.056334 / 0.023109 (0.033225) | 0.252281 / 0.275898 (-0.023617) | 0.274945 / 0.323480 (-0.048535) | 0.003906 / 0.007986 (-0.004080) | 0.002483 / 0.004328 (-0.001845) | 0.049006 / 0.004250 (0.044756) | 0.038375 / 0.037052 (0.001323) | 0.257376 / 0.258489 (-0.001113) | 0.292512 / 0.293841 (-0.001328) | 0.027134 / 0.128546 (-0.101412) | 0.010579 / 0.075646 (-0.065068) | 0.212021 / 0.419271 (-0.207250) | 0.035851 / 0.043533 (-0.007682) | 0.258076 / 0.255139 (0.002937) | 0.271758 / 0.283200 (-0.011442) | 0.018222 / 0.141683 (-0.123461) | 1.120481 / 1.452155 (-0.331674) | 1.187007 / 1.492716 (-0.305710) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094986 / 0.018006 (0.076980) | 0.302121 / 0.000490 (0.301631) | 0.000211 / 0.000200 (0.000011) | 0.000052 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019260 / 0.037411 (-0.018152) | 0.062909 / 0.014526 (0.048383) | 0.075644 / 0.176557 (-0.100912) | 0.120966 / 0.737135 (-0.616170) | 0.076678 / 0.296338 (-0.219661) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.286754 / 0.215209 (0.071545) | 2.797467 / 2.077655 (0.719812) | 1.436798 / 1.504120 (-0.067322) | 1.315032 / 1.541195 (-0.226163) | 1.367841 / 1.468490 (-0.100649) | 0.578917 / 4.584777 (-4.005860) | 2.439773 / 3.745712 (-1.305939) | 2.932779 / 5.269862 (-2.337082) | 1.843895 / 4.565676 (-2.721782) | 0.063351 / 0.424275 (-0.360925) | 0.004998 / 0.007607 (-0.002610) | 0.347385 / 0.226044 (0.121340) | 3.449969 / 2.268929 (1.181040) | 1.857734 / 55.444624 (-53.586890) | 1.541341 / 6.876477 (-5.335136) | 1.574915 / 2.142072 (-0.567158) | 0.660178 / 4.805227 (-4.145049) | 0.117686 / 6.500664 (-6.382978) | 0.042602 / 0.075469 (-0.032867) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.937735 / 1.841788 (-0.904052) | 11.962091 / 8.074308 (3.887783) | 10.401715 / 10.191392 (0.210323) | 0.142200 / 0.680424 (-0.538224) | 0.014137 / 0.534201 (-0.520064) | 0.289853 / 0.579283 (-0.289430) | 0.267100 / 0.434364 (-0.167264) | 0.323401 / 0.540337 (-0.216936) | 0.418665 / 1.386936 (-0.968271) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005480 / 0.011353 (-0.005873) | 0.003401 / 0.011008 (-0.007607) | 0.049304 / 0.038508 (0.010796) | 0.062043 / 0.023109 (0.038934) | 0.270571 / 0.275898 (-0.005327) | 0.295226 / 0.323480 (-0.028254) | 0.004152 / 0.007986 (-0.003834) | 0.002511 / 0.004328 (-0.001817) | 0.048480 / 0.004250 (0.044229) | 0.043964 / 0.037052 (0.006912) | 0.273545 / 0.258489 (0.015056) | 0.295152 / 0.293841 (0.001311) | 0.029224 / 0.128546 (-0.099322) | 0.010629 / 0.075646 (-0.065018) | 0.057433 / 0.419271 (-0.361839) | 0.033115 / 0.043533 (-0.010418) | 0.269893 / 0.255139 (0.014754) | 0.288658 / 0.283200 (0.005459) | 0.018216 / 0.141683 (-0.123467) | 1.123039 / 1.452155 (-0.329116) | 1.182892 / 1.492716 (-0.309825) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095948 / 0.018006 (0.077942) | 0.305811 / 0.000490 (0.305321) | 0.000221 / 0.000200 (0.000021) | 0.000053 / 0.000054 (-0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022996 / 0.037411 (-0.014415) | 0.073836 / 0.014526 (0.059310) | 0.082658 / 0.176557 (-0.093899) | 0.121970 / 0.737135 (-0.615166) | 0.086096 / 0.296338 (-0.210242) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.291032 / 0.215209 (0.075823) | 2.864613 / 2.077655 (0.786958) | 1.567530 / 1.504120 (0.063410) | 1.460291 / 1.541195 (-0.080903) | 1.527066 / 1.468490 (0.058576) | 0.571160 / 4.584777 (-4.013617) | 2.465261 / 3.745712 (-1.280451) | 2.915547 / 5.269862 (-2.354314) | 1.835822 / 4.565676 (-2.729855) | 0.064328 / 0.424275 (-0.359947) | 0.005061 / 0.007607 (-0.002546) | 0.357105 / 0.226044 (0.131061) | 3.491363 / 2.268929 (1.222435) | 1.943213 / 55.444624 (-53.501412) | 1.675778 / 6.876477 (-5.200699) | 1.719016 / 2.142072 (-0.423057) | 0.658993 / 4.805227 (-4.146235) | 0.122320 / 6.500664 (-6.378344) | 0.049030 / 0.075469 (-0.026439) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.964762 / 1.841788 (-0.877025) | 12.367251 / 8.074308 (4.292943) | 10.886213 / 10.191392 (0.694821) | 0.141533 / 0.680424 (-0.538891) | 0.015646 / 0.534201 (-0.518555) | 0.288583 / 0.579283 (-0.290700) | 0.280353 / 0.434364 (-0.154010) | 0.329095 / 0.540337 (-0.211242) | 0.565118 / 1.386936 (-0.821818) |

</details>
</details>

![](https://cml.dev/watermark.png#493bf695dc3ee6cc81bfd0aae6a38f70547bb752 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006475 / 0.011353 (-0.004878) | 0.004080 / 0.011008 (-0.006928) | 0.066479 / 0.038508 (0.027971) | 0.073270 / 0.023109 (0.050161) | 0.244412 / 0.275898 (-0.031486) | 0.273778 / 0.323480 (-0.049702) | 0.003186 / 0.007986 (-0.004800) | 0.003419 / 0.004328 (-0.000910) | 0.049743 / 0.004250 (0.045492) | 0.043581 / 0.037052 (0.006529) | 0.248215 / 0.258489 (-0.010274) | 0.280873 / 0.293841 (-0.012967) | 0.029282 / 0.128546 (-0.099264) | 0.011241 / 0.075646 (-0.064405) | 0.215031 / 0.419271 (-0.204241) | 0.038764 / 0.043533 (-0.004769) | 0.259363 / 0.255139 (0.004224) | 0.279253 / 0.283200 (-0.003946) | 0.019524 / 0.141683 (-0.122159) | 1.104735 / 1.452155 (-0.347420) | 1.159823 / 1.492716 (-0.332894) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.108383 / 0.018006 (0.090377) | 0.332904 / 0.000490 (0.332415) | 0.000222 / 0.000200 (0.000022) | 0.000065 / 0.000054 (0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.020693 / 0.037411 (-0.016719) | 0.071764 / 0.014526 (0.057238) | 0.077073 / 0.176557 (-0.099484) | 0.124604 / 0.737135 (-0.612532) | 0.078057 / 0.296338 (-0.218282) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.291014 / 0.215209 (0.075805) | 2.865885 / 2.077655 (0.788231) | 1.506141 / 1.504120 (0.002021) | 1.435924 / 1.541195 (-0.105271) | 1.461994 / 1.468490 (-0.006497) | 0.571779 / 4.584777 (-4.012998) | 2.461950 / 3.745712 (-1.283762) | 3.079771 / 5.269862 (-2.190091) | 1.933337 / 4.565676 (-2.632339) | 0.063405 / 0.424275 (-0.360870) | 0.005203 / 0.007607 (-0.002404) | 0.345077 / 0.226044 (0.119032) | 3.487189 / 2.268929 (1.218261) | 1.903733 / 55.444624 (-53.540891) | 1.705596 / 6.876477 (-5.170880) | 1.718849 / 2.142072 (-0.423223) | 0.658745 / 4.805227 (-4.146482) | 0.120847 / 6.500664 (-6.379817) | 0.045670 / 0.075469 (-0.029799) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.965969 / 1.841788 (-0.875819) | 13.520489 / 8.074308 (5.446181) | 12.322363 / 10.191392 (2.130971) | 0.146605 / 0.680424 (-0.533819) | 0.015061 / 0.534201 (-0.519140) | 0.298125 / 0.579283 (-0.281159) | 0.276864 / 0.434364 (-0.157500) | 0.326787 / 0.540337 (-0.213550) | 0.436897 / 1.386936 (-0.950039) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005862 / 0.011353 (-0.005491) | 0.003716 / 0.011008 (-0.007292) | 0.052849 / 0.038508 (0.014341) | 0.072114 / 0.023109 (0.049005) | 0.277800 / 0.275898 (0.001902) | 0.325321 / 0.323480 (0.001841) | 0.004428 / 0.007986 (-0.003557) | 0.002527 / 0.004328 (-0.001801) | 0.048847 / 0.004250 (0.044596) | 0.047355 / 0.037052 (0.010303) | 0.279331 / 0.258489 (0.020842) | 0.310477 / 0.293841 (0.016636) | 0.029661 / 0.128546 (-0.098886) | 0.010812 / 0.075646 (-0.064834) | 0.059803 / 0.419271 (-0.359469) | 0.033554 / 0.043533 (-0.009978) | 0.276890 / 0.255139 (0.021751) | 0.308911 / 0.283200 (0.025712) | 0.020752 / 0.141683 (-0.120931) | 1.120896 / 1.452155 (-0.331259) | 1.186428 / 1.492716 (-0.306288) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.106551 / 0.018006 (0.088545) | 0.354455 / 0.000490 (0.353966) | 0.000353 / 0.000200 (0.000153) | 0.000069 / 0.000054 (0.000015) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023488 / 0.037411 (-0.013923) | 0.080548 / 0.014526 (0.066022) | 0.084431 / 0.176557 (-0.092126) | 0.140698 / 0.737135 (-0.596438) | 0.085692 / 0.296338 (-0.210647) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.314253 / 0.215209 (0.099044) | 2.993236 / 2.077655 (0.915582) | 1.639013 / 1.504120 (0.134893) | 1.543966 / 1.541195 (0.002771) | 1.567732 / 1.468490 (0.099242) | 0.565857 / 4.584777 (-4.018920) | 2.545339 / 3.745712 (-1.200373) | 3.134546 / 5.269862 (-2.135316) | 1.940350 / 4.565676 (-2.625326) | 0.063847 / 0.424275 (-0.360429) | 0.005079 / 0.007607 (-0.002528) | 0.365762 / 0.226044 (0.139718) | 3.610921 / 2.268929 (1.341993) | 2.035151 / 55.444624 (-53.409473) | 1.773409 / 6.876477 (-5.103068) | 1.790332 / 2.142072 (-0.351741) | 0.683019 / 4.805227 (-4.122209) | 0.119566 / 6.500664 (-6.381099) | 0.043578 / 0.075469 (-0.031891) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.996568 / 1.841788 (-0.845219) | 14.094366 / 8.074308 (6.020058) | 12.433600 / 10.191392 (2.242208) | 0.139835 / 0.680424 (-0.540589) | 0.016454 / 0.534201 (-0.517747) | 0.294073 / 0.579283 (-0.285210) | 0.309032 / 0.434364 (-0.125332) | 0.330699 / 0.540337 (-0.209638) | 0.619392 / 1.386936 (-0.767544) |

</details>
</details>

![](https://cml.dev/watermark.png#026fbce1c93a30188b6d0646bb975da8f56e2a2f "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005389 / 0.011353 (-0.005964) | 0.003209 / 0.011008 (-0.007799) | 0.061610 / 0.038508 (0.023102) | 0.049781 / 0.023109 (0.026672) | 0.240208 / 0.275898 (-0.035690) | 0.263307 / 0.323480 (-0.060173) | 0.002908 / 0.007986 (-0.005078) | 0.002375 / 0.004328 (-0.001953) | 0.047462 / 0.004250 (0.043212) | 0.038643 / 0.037052 (0.001591) | 0.246287 / 0.258489 (-0.012202) | 0.278715 / 0.293841 (-0.015126) | 0.027507 / 0.128546 (-0.101039) | 0.010168 / 0.075646 (-0.065479) | 0.204131 / 0.419271 (-0.215140) | 0.035452 / 0.043533 (-0.008081) | 0.251721 / 0.255139 (-0.003418) | 0.266642 / 0.283200 (-0.016558) | 0.017741 / 0.141683 (-0.123942) | 1.094672 / 1.452155 (-0.357482) | 1.162715 / 1.492716 (-0.330002) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092154 / 0.018006 (0.074148) | 0.301376 / 0.000490 (0.300886) | 0.000217 / 0.000200 (0.000017) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018534 / 0.037411 (-0.018877) | 0.061995 / 0.014526 (0.047469) | 0.072654 / 0.176557 (-0.103903) | 0.119501 / 0.737135 (-0.617635) | 0.073756 / 0.296338 (-0.222583) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.280066 / 0.215209 (0.064857) | 2.744207 / 2.077655 (0.666553) | 1.483367 / 1.504120 (-0.020753) | 1.386173 / 1.541195 (-0.155022) | 1.381833 / 1.468490 (-0.086657) | 0.552780 / 4.584777 (-4.031997) | 2.395541 / 3.745712 (-1.350171) | 2.747507 / 5.269862 (-2.522355) | 1.735074 / 4.565676 (-2.830602) | 0.062096 / 0.424275 (-0.362179) | 0.004905 / 0.007607 (-0.002702) | 0.338327 / 0.226044 (0.112283) | 3.365391 / 2.268929 (1.096462) | 1.839663 / 55.444624 (-53.604961) | 1.577535 / 6.876477 (-5.298942) | 1.558054 / 2.142072 (-0.584018) | 0.636520 / 4.805227 (-4.168708) | 0.116182 / 6.500664 (-6.384482) | 0.042078 / 0.075469 (-0.033391) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.938512 / 1.841788 (-0.903276) | 11.455749 / 8.074308 (3.381441) | 10.510985 / 10.191392 (0.319593) | 0.140865 / 0.680424 (-0.539559) | 0.014073 / 0.534201 (-0.520128) | 0.294747 / 0.579283 (-0.284536) | 0.266147 / 0.434364 (-0.168217) | 0.325354 / 0.540337 (-0.214984) | 0.422182 / 1.386936 (-0.964754) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005231 / 0.011353 (-0.006122) | 0.003032 / 0.011008 (-0.007977) | 0.049608 / 0.038508 (0.011099) | 0.051441 / 0.023109 (0.028332) | 0.273812 / 0.275898 (-0.002086) | 0.294318 / 0.323480 (-0.029162) | 0.003958 / 0.007986 (-0.004028) | 0.002384 / 0.004328 (-0.001944) | 0.047942 / 0.004250 (0.043691) | 0.039179 / 0.037052 (0.002127) | 0.277504 / 0.258489 (0.019014) | 0.299713 / 0.293841 (0.005872) | 0.028989 / 0.128546 (-0.099557) | 0.010267 / 0.075646 (-0.065379) | 0.058318 / 0.419271 (-0.360954) | 0.032214 / 0.043533 (-0.011318) | 0.277964 / 0.255139 (0.022825) | 0.293055 / 0.283200 (0.009856) | 0.018532 / 0.141683 (-0.123151) | 1.128620 / 1.452155 (-0.323535) | 1.187365 / 1.492716 (-0.305351) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092137 / 0.018006 (0.074130) | 0.299726 / 0.000490 (0.299236) | 0.000222 / 0.000200 (0.000022) | 0.000050 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021342 / 0.037411 (-0.016070) | 0.069943 / 0.014526 (0.055417) | 0.079862 / 0.176557 (-0.096694) | 0.118917 / 0.737135 (-0.618218) | 0.081861 / 0.296338 (-0.214477) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.295883 / 0.215209 (0.080674) | 2.881640 / 2.077655 (0.803986) | 1.597705 / 1.504120 (0.093585) | 1.473220 / 1.541195 (-0.067975) | 1.501006 / 1.468490 (0.032516) | 0.559409 / 4.584777 (-4.025368) | 2.442709 / 3.745712 (-1.303003) | 2.742139 / 5.269862 (-2.527723) | 1.726002 / 4.565676 (-2.839674) | 0.062436 / 0.424275 (-0.361840) | 0.004896 / 0.007607 (-0.002711) | 0.349203 / 0.226044 (0.123159) | 3.435175 / 2.268929 (1.166247) | 1.954888 / 55.444624 (-53.489737) | 1.666233 / 6.876477 (-5.210243) | 1.680852 / 2.142072 (-0.461221) | 0.644271 / 4.805227 (-4.160956) | 0.115160 / 6.500664 (-6.385504) | 0.040681 / 0.075469 (-0.034788) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.963810 / 1.841788 (-0.877977) | 11.860860 / 8.074308 (3.786552) | 10.541703 / 10.191392 (0.350311) | 0.131532 / 0.680424 (-0.548892) | 0.016790 / 0.534201 (-0.517411) | 0.286695 / 0.579283 (-0.292588) | 0.279628 / 0.434364 (-0.154735) | 0.324622 / 0.540337 (-0.215715) | 0.535507 / 1.386936 (-0.851429) |

</details>
</details>

![](https://cml.dev/watermark.png#11217347e4bcfe1aaf794d164a5dd9f085b2f682 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005672 / 0.011353 (-0.005681) | 0.003411 / 0.011008 (-0.007597) | 0.062528 / 0.038508 (0.024020) | 0.055209 / 0.023109 (0.032100) | 0.248366 / 0.275898 (-0.027532) | 0.279522 / 0.323480 (-0.043957) | 0.002907 / 0.007986 (-0.005079) | 0.002369 / 0.004328 (-0.001959) | 0.047982 / 0.004250 (0.043731) | 0.039009 / 0.037052 (0.001956) | 0.256422 / 0.258489 (-0.002067) | 0.288530 / 0.293841 (-0.005311) | 0.028164 / 0.128546 (-0.100382) | 0.010448 / 0.075646 (-0.065198) | 0.208863 / 0.419271 (-0.210408) | 0.036291 / 0.043533 (-0.007242) | 0.251642 / 0.255139 (-0.003497) | 0.275589 / 0.283200 (-0.007610) | 0.019839 / 0.141683 (-0.121844) | 1.092800 / 1.452155 (-0.359355) | 1.147950 / 1.492716 (-0.344766) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094920 / 0.018006 (0.076914) | 0.303049 / 0.000490 (0.302559) | 0.000199 / 0.000200 (-0.000001) | 0.000043 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018820 / 0.037411 (-0.018591) | 0.063319 / 0.014526 (0.048793) | 0.073644 / 0.176557 (-0.102912) | 0.120045 / 0.737135 (-0.617091) | 0.076219 / 0.296338 (-0.220119) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.283897 / 0.215209 (0.068688) | 2.822836 / 2.077655 (0.745182) | 1.490505 / 1.504120 (-0.013615) | 1.359777 / 1.541195 (-0.181418) | 1.420536 / 1.468490 (-0.047954) | 0.562308 / 4.584777 (-4.022469) | 2.419249 / 3.745712 (-1.326463) | 2.827620 / 5.269862 (-2.442241) | 1.783171 / 4.565676 (-2.782505) | 0.063206 / 0.424275 (-0.361069) | 0.004966 / 0.007607 (-0.002641) | 0.339647 / 0.226044 (0.113602) | 3.378157 / 2.268929 (1.109229) | 1.873221 / 55.444624 (-53.571403) | 1.606367 / 6.876477 (-5.270109) | 1.624976 / 2.142072 (-0.517096) | 0.652653 / 4.805227 (-4.152574) | 0.117997 / 6.500664 (-6.382667) | 0.041955 / 0.075469 (-0.033514) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.961420 / 1.841788 (-0.880368) | 11.807624 / 8.074308 (3.733316) | 10.668249 / 10.191392 (0.476857) | 0.141855 / 0.680424 (-0.538569) | 0.014451 / 0.534201 (-0.519750) | 0.289706 / 0.579283 (-0.289577) | 0.268392 / 0.434364 (-0.165972) | 0.323435 / 0.540337 (-0.216903) | 0.420667 / 1.386936 (-0.966269) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005382 / 0.011353 (-0.005971) | 0.003361 / 0.011008 (-0.007647) | 0.048420 / 0.038508 (0.009912) | 0.053702 / 0.023109 (0.030593) | 0.286976 / 0.275898 (0.011078) | 0.296708 / 0.323480 (-0.026772) | 0.004013 / 0.007986 (-0.003972) | 0.002444 / 0.004328 (-0.001884) | 0.047797 / 0.004250 (0.043547) | 0.042361 / 0.037052 (0.005309) | 0.277543 / 0.258489 (0.019054) | 0.300736 / 0.293841 (0.006896) | 0.029894 / 0.128546 (-0.098653) | 0.014119 / 0.075646 (-0.061527) | 0.057636 / 0.419271 (-0.361636) | 0.032533 / 0.043533 (-0.010999) | 0.280963 / 0.255139 (0.025824) | 0.291305 / 0.283200 (0.008106) | 0.018391 / 0.141683 (-0.123292) | 1.140042 / 1.452155 (-0.312113) | 1.179485 / 1.492716 (-0.313231) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094668 / 0.018006 (0.076661) | 0.301677 / 0.000490 (0.301187) | 0.000245 / 0.000200 (0.000045) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021376 / 0.037411 (-0.016036) | 0.070628 / 0.014526 (0.056102) | 0.082249 / 0.176557 (-0.094308) | 0.120423 / 0.737135 (-0.616712) | 0.083792 / 0.296338 (-0.212546) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.298884 / 0.215209 (0.083675) | 2.931849 / 2.077655 (0.854194) | 1.591888 / 1.504120 (0.087768) | 1.455781 / 1.541195 (-0.085414) | 1.500312 / 1.468490 (0.031822) | 0.558466 / 4.584777 (-4.026311) | 2.450449 / 3.745712 (-1.295263) | 2.842768 / 5.269862 (-2.427094) | 1.755614 / 4.565676 (-2.810062) | 0.063200 / 0.424275 (-0.361075) | 0.005022 / 0.007607 (-0.002585) | 0.358282 / 0.226044 (0.132238) | 3.575392 / 2.268929 (1.306464) | 1.960258 / 55.444624 (-53.484366) | 1.675518 / 6.876477 (-5.200959) | 1.696630 / 2.142072 (-0.445442) | 0.647185 / 4.805227 (-4.158042) | 0.117038 / 6.500664 (-6.383626) | 0.041622 / 0.075469 (-0.033848) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.962503 / 1.841788 (-0.879285) | 12.194950 / 8.074308 (4.120642) | 10.662233 / 10.191392 (0.470841) | 0.131618 / 0.680424 (-0.548806) | 0.016000 / 0.534201 (-0.518201) | 0.291546 / 0.579283 (-0.287737) | 0.279537 / 0.434364 (-0.154827) | 0.328716 / 0.540337 (-0.211622) | 0.547565 / 1.386936 (-0.839371) |

</details>
</details>

![](https://cml.dev/watermark.png#4de8f5f09f60613d47b5d7eb901752321c7b6a49 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005209 / 0.011353 (-0.006144) | 0.003017 / 0.011008 (-0.007991) | 0.062017 / 0.038508 (0.023509) | 0.048268 / 0.023109 (0.025158) | 0.246384 / 0.275898 (-0.029514) | 0.270441 / 0.323480 (-0.053039) | 0.002763 / 0.007986 (-0.005222) | 0.003140 / 0.004328 (-0.001188) | 0.048720 / 0.004250 (0.044470) | 0.038175 / 0.037052 (0.001123) | 0.254184 / 0.258489 (-0.004306) | 0.275515 / 0.293841 (-0.018326) | 0.027309 / 0.128546 (-0.101238) | 0.010507 / 0.075646 (-0.065140) | 0.210315 / 0.419271 (-0.208956) | 0.035203 / 0.043533 (-0.008329) | 0.253015 / 0.255139 (-0.002124) | 0.271465 / 0.283200 (-0.011734) | 0.019543 / 0.141683 (-0.122140) | 1.119242 / 1.452155 (-0.332913) | 1.149359 / 1.492716 (-0.343357) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.088935 / 0.018006 (0.070928) | 0.293922 / 0.000490 (0.293432) | 0.000202 / 0.000200 (0.000002) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018174 / 0.037411 (-0.019237) | 0.060215 / 0.014526 (0.045689) | 0.072868 / 0.176557 (-0.103689) | 0.117998 / 0.737135 (-0.619137) | 0.074159 / 0.296338 (-0.222179) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.289229 / 0.215209 (0.074020) | 2.840414 / 2.077655 (0.762759) | 1.468357 / 1.504120 (-0.035763) | 1.347714 / 1.541195 (-0.193481) | 1.363704 / 1.468490 (-0.104786) | 0.572059 / 4.584777 (-4.012718) | 2.400631 / 3.745712 (-1.345081) | 2.755779 / 5.269862 (-2.514083) | 1.740937 / 4.565676 (-2.824739) | 0.063473 / 0.424275 (-0.360802) | 0.005012 / 0.007607 (-0.002595) | 0.336057 / 0.226044 (0.110012) | 3.382126 / 2.268929 (1.113197) | 1.807838 / 55.444624 (-53.636786) | 1.534594 / 6.876477 (-5.341883) | 1.529951 / 2.142072 (-0.612121) | 0.636661 / 4.805227 (-4.168566) | 0.117090 / 6.500664 (-6.383574) | 0.042310 / 0.075469 (-0.033160) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.924440 / 1.841788 (-0.917347) | 11.120517 / 8.074308 (3.046209) | 10.177210 / 10.191392 (-0.014182) | 0.139060 / 0.680424 (-0.541364) | 0.013818 / 0.534201 (-0.520383) | 0.285634 / 0.579283 (-0.293649) | 0.268657 / 0.434364 (-0.165706) | 0.325842 / 0.540337 (-0.214496) | 0.439902 / 1.386936 (-0.947034) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005202 / 0.011353 (-0.006150) | 0.003002 / 0.011008 (-0.008006) | 0.048729 / 0.038508 (0.010221) | 0.048178 / 0.023109 (0.025069) | 0.288573 / 0.275898 (0.012675) | 0.311122 / 0.323480 (-0.012358) | 0.003953 / 0.007986 (-0.004033) | 0.002544 / 0.004328 (-0.001785) | 0.047762 / 0.004250 (0.043511) | 0.039711 / 0.037052 (0.002658) | 0.308389 / 0.258489 (0.049900) | 0.321913 / 0.293841 (0.028072) | 0.029166 / 0.128546 (-0.099380) | 0.010697 / 0.075646 (-0.064950) | 0.057758 / 0.419271 (-0.361514) | 0.032743 / 0.043533 (-0.010789) | 0.290933 / 0.255139 (0.035794) | 0.309404 / 0.283200 (0.026205) | 0.017691 / 0.141683 (-0.123992) | 1.157713 / 1.452155 (-0.294442) | 1.210485 / 1.492716 (-0.282231) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.088959 / 0.018006 (0.070953) | 0.298531 / 0.000490 (0.298041) | 0.000221 / 0.000200 (0.000021) | 0.000053 / 0.000054 (-0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021129 / 0.037411 (-0.016283) | 0.068419 / 0.014526 (0.053893) | 0.079328 / 0.176557 (-0.097228) | 0.118603 / 0.737135 (-0.618532) | 0.080489 / 0.296338 (-0.215850) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.292464 / 0.215209 (0.077254) | 2.898221 / 2.077655 (0.820566) | 1.600868 / 1.504120 (0.096748) | 1.485128 / 1.541195 (-0.056067) | 1.493091 / 1.468490 (0.024600) | 0.576117 / 4.584777 (-4.008660) | 2.450440 / 3.745712 (-1.295273) | 2.746026 / 5.269862 (-2.523836) | 1.722555 / 4.565676 (-2.843122) | 0.062869 / 0.424275 (-0.361406) | 0.004918 / 0.007607 (-0.002689) | 0.348470 / 0.226044 (0.122425) | 3.420267 / 2.268929 (1.151339) | 1.942973 / 55.444624 (-53.501651) | 1.667684 / 6.876477 (-5.208793) | 1.669618 / 2.142072 (-0.472454) | 0.630275 / 4.805227 (-4.174952) | 0.115072 / 6.500664 (-6.385592) | 0.040430 / 0.075469 (-0.035039) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.989827 / 1.841788 (-0.851961) | 11.578068 / 8.074308 (3.503760) | 10.636060 / 10.191392 (0.444668) | 0.131943 / 0.680424 (-0.548481) | 0.015915 / 0.534201 (-0.518286) | 0.287277 / 0.579283 (-0.292006) | 0.279451 / 0.434364 (-0.154913) | 0.325485 / 0.540337 (-0.214852) | 0.544635 / 1.386936 (-0.842301) |

</details>
</details>

![](https://cml.dev/watermark.png#f22579be6c73867ac1a3c03e925abaf4872f8437 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005144 / 0.011353 (-0.006209) | 0.003686 / 0.011008 (-0.007322) | 0.064003 / 0.038508 (0.025495) | 0.058962 / 0.023109 (0.035853) | 0.233753 / 0.275898 (-0.042145) | 0.255802 / 0.323480 (-0.067677) | 0.003871 / 0.007986 (-0.004115) | 0.002609 / 0.004328 (-0.001719) | 0.048675 / 0.004250 (0.044425) | 0.037550 / 0.037052 (0.000498) | 0.240658 / 0.258489 (-0.017831) | 0.272303 / 0.293841 (-0.021538) | 0.027455 / 0.128546 (-0.101091) | 0.010706 / 0.075646 (-0.064941) | 0.210878 / 0.419271 (-0.208393) | 0.035763 / 0.043533 (-0.007770) | 0.239937 / 0.255139 (-0.015202) | 0.262520 / 0.283200 (-0.020680) | 0.017676 / 0.141683 (-0.124006) | 1.095036 / 1.452155 (-0.357118) | 1.178318 / 1.492716 (-0.314399) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095310 / 0.018006 (0.077304) | 0.307485 / 0.000490 (0.306995) | 0.000212 / 0.000200 (0.000013) | 0.000047 / 0.000054 (-0.000007) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018630 / 0.037411 (-0.018781) | 0.060461 / 0.014526 (0.045936) | 0.073117 / 0.176557 (-0.103440) | 0.119737 / 0.737135 (-0.617399) | 0.073909 / 0.296338 (-0.222430) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.280938 / 0.215209 (0.065729) | 2.755333 / 2.077655 (0.677679) | 1.468153 / 1.504120 (-0.035967) | 1.350247 / 1.541195 (-0.190948) | 1.379834 / 1.468490 (-0.088656) | 0.564027 / 4.584777 (-4.020750) | 2.387794 / 3.745712 (-1.357918) | 2.768529 / 5.269862 (-2.501333) | 1.761994 / 4.565676 (-2.803682) | 0.062079 / 0.424275 (-0.362196) | 0.005018 / 0.007607 (-0.002589) | 0.337576 / 0.226044 (0.111532) | 3.345347 / 2.268929 (1.076418) | 1.821950 / 55.444624 (-53.622674) | 1.545471 / 6.876477 (-5.331006) | 1.534941 / 2.142072 (-0.607131) | 0.626560 / 4.805227 (-4.178668) | 0.116227 / 6.500664 (-6.384437) | 0.041722 / 0.075469 (-0.033747) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.950480 / 1.841788 (-0.891307) | 11.616355 / 8.074308 (3.542047) | 10.426687 / 10.191392 (0.235295) | 0.129967 / 0.680424 (-0.550457) | 0.013977 / 0.534201 (-0.520224) | 0.287150 / 0.579283 (-0.292133) | 0.264028 / 0.434364 (-0.170336) | 0.325061 / 0.540337 (-0.215277) | 0.441281 / 1.386936 (-0.945655) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005436 / 0.011353 (-0.005917) | 0.003567 / 0.011008 (-0.007441) | 0.055275 / 0.038508 (0.016767) | 0.053216 / 0.023109 (0.030107) | 0.272826 / 0.275898 (-0.003072) | 0.298399 / 0.323480 (-0.025081) | 0.004803 / 0.007986 (-0.003183) | 0.002681 / 0.004328 (-0.001648) | 0.048704 / 0.004250 (0.044453) | 0.040048 / 0.037052 (0.002996) | 0.278200 / 0.258489 (0.019711) | 0.331167 / 0.293841 (0.037326) | 0.029282 / 0.128546 (-0.099265) | 0.010766 / 0.075646 (-0.064881) | 0.057370 / 0.419271 (-0.361902) | 0.032674 / 0.043533 (-0.010859) | 0.269430 / 0.255139 (0.014291) | 0.288256 / 0.283200 (0.005056) | 0.019340 / 0.141683 (-0.122343) | 1.118058 / 1.452155 (-0.334097) | 1.157811 / 1.492716 (-0.334906) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094091 / 0.018006 (0.076085) | 0.301833 / 0.000490 (0.301343) | 0.000216 / 0.000200 (0.000016) | 0.000053 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021327 / 0.037411 (-0.016085) | 0.068636 / 0.014526 (0.054110) | 0.080246 / 0.176557 (-0.096311) | 0.120524 / 0.737135 (-0.616611) | 0.082226 / 0.296338 (-0.214113) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.293579 / 0.215209 (0.078370) | 2.880281 / 2.077655 (0.802626) | 1.594647 / 1.504120 (0.090528) | 1.477152 / 1.541195 (-0.064043) | 1.498122 / 1.468490 (0.029632) | 0.555073 / 4.584777 (-4.029704) | 2.446743 / 3.745712 (-1.298970) | 2.794971 / 5.269862 (-2.474890) | 1.749730 / 4.565676 (-2.815947) | 0.062537 / 0.424275 (-0.361738) | 0.004908 / 0.007607 (-0.002699) | 0.350772 / 0.226044 (0.124727) | 3.486535 / 2.268929 (1.217607) | 1.957414 / 55.444624 (-53.487210) | 1.669169 / 6.876477 (-5.207308) | 1.682396 / 2.142072 (-0.459676) | 0.627379 / 4.805227 (-4.177848) | 0.117218 / 6.500664 (-6.383446) | 0.041000 / 0.075469 (-0.034469) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.958248 / 1.841788 (-0.883539) | 12.022677 / 8.074308 (3.948369) | 10.331661 / 10.191392 (0.140269) | 0.129765 / 0.680424 (-0.550659) | 0.015073 / 0.534201 (-0.519128) | 0.287212 / 0.579283 (-0.292071) | 0.278310 / 0.434364 (-0.156054) | 0.328155 / 0.540337 (-0.212183) | 0.564990 / 1.386936 (-0.821946) |

</details>
</details>

![](https://cml.dev/watermark.png#0c16e56371e50adae771288945e3389cb81a31fd "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005576 / 0.011353 (-0.005777) | 0.003430 / 0.011008 (-0.007578) | 0.062714 / 0.038508 (0.024206) | 0.051240 / 0.023109 (0.028131) | 0.236637 / 0.275898 (-0.039261) | 0.262660 / 0.323480 (-0.060820) | 0.002924 / 0.007986 (-0.005061) | 0.002712 / 0.004328 (-0.001616) | 0.048680 / 0.004250 (0.044430) | 0.038997 / 0.037052 (0.001945) | 0.241426 / 0.258489 (-0.017063) | 0.270652 / 0.293841 (-0.023189) | 0.027355 / 0.128546 (-0.101192) | 0.010640 / 0.075646 (-0.065006) | 0.207754 / 0.419271 (-0.211517) | 0.035921 / 0.043533 (-0.007612) | 0.247645 / 0.255139 (-0.007494) | 0.262933 / 0.283200 (-0.020266) | 0.019658 / 0.141683 (-0.122025) | 1.112576 / 1.452155 (-0.339578) | 1.177362 / 1.492716 (-0.315354) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.098100 / 0.018006 (0.080093) | 0.310170 / 0.000490 (0.309680) | 0.000220 / 0.000200 (0.000020) | 0.000051 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019626 / 0.037411 (-0.017785) | 0.065468 / 0.014526 (0.050942) | 0.074767 / 0.176557 (-0.101789) | 0.123619 / 0.737135 (-0.613516) | 0.077159 / 0.296338 (-0.219179) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.288585 / 0.215209 (0.073376) | 2.771254 / 2.077655 (0.693599) | 1.457091 / 1.504120 (-0.047029) | 1.324341 / 1.541195 (-0.216854) | 1.361960 / 1.468490 (-0.106530) | 0.574197 / 4.584777 (-4.010580) | 2.391440 / 3.745712 (-1.354273) | 2.935060 / 5.269862 (-2.334802) | 1.802792 / 4.565676 (-2.762884) | 0.063530 / 0.424275 (-0.360745) | 0.005129 / 0.007607 (-0.002478) | 0.345977 / 0.226044 (0.119933) | 3.368042 / 2.268929 (1.099113) | 1.789575 / 55.444624 (-53.655050) | 1.509165 / 6.876477 (-5.367312) | 1.579792 / 2.142072 (-0.562280) | 0.652136 / 4.805227 (-4.153091) | 0.117014 / 6.500664 (-6.383650) | 0.042385 / 0.075469 (-0.033084) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.963967 / 1.841788 (-0.877821) | 11.847856 / 8.074308 (3.773548) | 10.584088 / 10.191392 (0.392696) | 0.143953 / 0.680424 (-0.536471) | 0.014355 / 0.534201 (-0.519846) | 0.286936 / 0.579283 (-0.292347) | 0.269039 / 0.434364 (-0.165325) | 0.324531 / 0.540337 (-0.215807) | 0.443187 / 1.386936 (-0.943749) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005448 / 0.011353 (-0.005905) | 0.003742 / 0.011008 (-0.007266) | 0.048808 / 0.038508 (0.010300) | 0.055409 / 0.023109 (0.032300) | 0.271574 / 0.275898 (-0.004324) | 0.295599 / 0.323480 (-0.027881) | 0.004208 / 0.007986 (-0.003778) | 0.002683 / 0.004328 (-0.001645) | 0.048813 / 0.004250 (0.044562) | 0.043672 / 0.037052 (0.006620) | 0.282173 / 0.258489 (0.023684) | 0.295447 / 0.293841 (0.001606) | 0.030461 / 0.128546 (-0.098086) | 0.010988 / 0.075646 (-0.064658) | 0.057050 / 0.419271 (-0.362221) | 0.033329 / 0.043533 (-0.010203) | 0.269700 / 0.255139 (0.014561) | 0.287099 / 0.283200 (0.003899) | 0.018203 / 0.141683 (-0.123480) | 1.142584 / 1.452155 (-0.309571) | 1.181848 / 1.492716 (-0.310869) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096958 / 0.018006 (0.078952) | 0.310563 / 0.000490 (0.310074) | 0.000224 / 0.000200 (0.000024) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022213 / 0.037411 (-0.015199) | 0.072054 / 0.014526 (0.057528) | 0.086393 / 0.176557 (-0.090163) | 0.122431 / 0.737135 (-0.614704) | 0.085298 / 0.296338 (-0.211041) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.290823 / 0.215209 (0.075614) | 2.838026 / 2.077655 (0.760371) | 1.541425 / 1.504120 (0.037305) | 1.431903 / 1.541195 (-0.109292) | 1.476567 / 1.468490 (0.008077) | 0.557856 / 4.584777 (-4.026920) | 2.449101 / 3.745712 (-1.296611) | 2.924633 / 5.269862 (-2.345229) | 1.824420 / 4.565676 (-2.741256) | 0.063735 / 0.424275 (-0.360540) | 0.005025 / 0.007607 (-0.002582) | 0.349458 / 0.226044 (0.123413) | 3.468627 / 2.268929 (1.199699) | 1.925173 / 55.444624 (-53.519451) | 1.655038 / 6.876477 (-5.221439) | 1.698612 / 2.142072 (-0.443460) | 0.643623 / 4.805227 (-4.161604) | 0.116128 / 6.500664 (-6.384536) | 0.042283 / 0.075469 (-0.033186) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.963029 / 1.841788 (-0.878758) | 13.273985 / 8.074308 (5.199677) | 11.400884 / 10.191392 (1.209492) | 0.152635 / 0.680424 (-0.527788) | 0.016442 / 0.534201 (-0.517759) | 0.289272 / 0.579283 (-0.290012) | 0.285286 / 0.434364 (-0.149078) | 0.330028 / 0.540337 (-0.210310) | 0.596500 / 1.386936 (-0.790436) |

</details>
</details>

![](https://cml.dev/watermark.png#9c427c4b1dcf84c898ae62dc521bf446bb35e0e7 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005124 / 0.011353 (-0.006229) | 0.003832 / 0.011008 (-0.007176) | 0.062806 / 0.038508 (0.024298) | 0.053137 / 0.023109 (0.030028) | 0.241155 / 0.275898 (-0.034743) | 0.260521 / 0.323480 (-0.062959) | 0.004005 / 0.007986 (-0.003981) | 0.002754 / 0.004328 (-0.001575) | 0.048934 / 0.004250 (0.044684) | 0.039438 / 0.037052 (0.002385) | 0.242534 / 0.258489 (-0.015955) | 0.275498 / 0.293841 (-0.018343) | 0.027338 / 0.128546 (-0.101208) | 0.010809 / 0.075646 (-0.064837) | 0.206986 / 0.419271 (-0.212285) | 0.035614 / 0.043533 (-0.007919) | 0.245780 / 0.255139 (-0.009359) | 0.259793 / 0.283200 (-0.023407) | 0.018108 / 0.141683 (-0.123575) | 1.103412 / 1.452155 (-0.348742) | 1.162940 / 1.492716 (-0.329776) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092463 / 0.018006 (0.074457) | 0.299516 / 0.000490 (0.299026) | 0.000210 / 0.000200 (0.000010) | 0.000047 / 0.000054 (-0.000007) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018261 / 0.037411 (-0.019150) | 0.060178 / 0.014526 (0.045652) | 0.073043 / 0.176557 (-0.103513) | 0.120541 / 0.737135 (-0.616594) | 0.074972 / 0.296338 (-0.221367) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.287288 / 0.215209 (0.072078) | 2.814915 / 2.077655 (0.737260) | 1.520221 / 1.504120 (0.016101) | 1.396045 / 1.541195 (-0.145149) | 1.419662 / 1.468490 (-0.048828) | 0.589247 / 4.584777 (-3.995530) | 2.411101 / 3.745712 (-1.334611) | 2.777709 / 5.269862 (-2.492153) | 1.750386 / 4.565676 (-2.815291) | 0.063734 / 0.424275 (-0.360541) | 0.005021 / 0.007607 (-0.002586) | 0.338817 / 0.226044 (0.112773) | 3.371218 / 2.268929 (1.102289) | 1.892691 / 55.444624 (-53.551934) | 1.599039 / 6.876477 (-5.277438) | 1.574726 / 2.142072 (-0.567346) | 0.665623 / 4.805227 (-4.139604) | 0.118628 / 6.500664 (-6.382036) | 0.041803 / 0.075469 (-0.033666) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.948696 / 1.841788 (-0.893092) | 11.502916 / 8.074308 (3.428608) | 10.301174 / 10.191392 (0.109782) | 0.141752 / 0.680424 (-0.538672) | 0.014064 / 0.534201 (-0.520137) | 0.286701 / 0.579283 (-0.292583) | 0.265805 / 0.434364 (-0.168559) | 0.328420 / 0.540337 (-0.211917) | 0.433619 / 1.386936 (-0.953317) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005262 / 0.011353 (-0.006091) | 0.003361 / 0.011008 (-0.007648) | 0.049525 / 0.038508 (0.011016) | 0.048950 / 0.023109 (0.025841) | 0.273617 / 0.275898 (-0.002281) | 0.296614 / 0.323480 (-0.026866) | 0.004014 / 0.007986 (-0.003971) | 0.002630 / 0.004328 (-0.001698) | 0.048203 / 0.004250 (0.043952) | 0.040912 / 0.037052 (0.003860) | 0.279736 / 0.258489 (0.021247) | 0.301671 / 0.293841 (0.007830) | 0.028546 / 0.128546 (-0.100000) | 0.010440 / 0.075646 (-0.065206) | 0.057869 / 0.419271 (-0.361402) | 0.032876 / 0.043533 (-0.010657) | 0.277649 / 0.255139 (0.022510) | 0.296565 / 0.283200 (0.013365) | 0.017558 / 0.141683 (-0.124125) | 1.155005 / 1.452155 (-0.297149) | 1.204827 / 1.492716 (-0.287889) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093248 / 0.018006 (0.075242) | 0.302721 / 0.000490 (0.302231) | 0.000218 / 0.000200 (0.000018) | 0.000048 / 0.000054 (-0.000006) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021882 / 0.037411 (-0.015530) | 0.068259 / 0.014526 (0.053733) | 0.080982 / 0.176557 (-0.095574) | 0.119386 / 0.737135 (-0.617750) | 0.081745 / 0.296338 (-0.214593) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.297812 / 0.215209 (0.082603) | 2.909938 / 2.077655 (0.832283) | 1.603736 / 1.504120 (0.099616) | 1.482989 / 1.541195 (-0.058206) | 1.495107 / 1.468490 (0.026617) | 0.562275 / 4.584777 (-4.022502) | 2.424812 / 3.745712 (-1.320901) | 2.759127 / 5.269862 (-2.510735) | 1.733283 / 4.565676 (-2.832394) | 0.063144 / 0.424275 (-0.361131) | 0.004949 / 0.007607 (-0.002658) | 0.352756 / 0.226044 (0.126711) | 3.496028 / 2.268929 (1.227100) | 1.982804 / 55.444624 (-53.461820) | 1.689787 / 6.876477 (-5.186690) | 1.672699 / 2.142072 (-0.469373) | 0.660169 / 4.805227 (-4.145059) | 0.116535 / 6.500664 (-6.384129) | 0.040616 / 0.075469 (-0.034853) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.975055 / 1.841788 (-0.866733) | 11.919295 / 8.074308 (3.844986) | 10.779188 / 10.191392 (0.587796) | 0.143106 / 0.680424 (-0.537318) | 0.015159 / 0.534201 (-0.519041) | 0.289734 / 0.579283 (-0.289549) | 0.278637 / 0.434364 (-0.155727) | 0.328159 / 0.540337 (-0.212178) | 0.570560 / 1.386936 (-0.816376) |

</details>
</details>

![](https://cml.dev/watermark.png#241500208da5fef64ad6ddc1cc5ab2be18f2f76d "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005155 / 0.011353 (-0.006198) | 0.003589 / 0.011008 (-0.007419) | 0.064440 / 0.038508 (0.025932) | 0.051020 / 0.023109 (0.027911) | 0.246099 / 0.275898 (-0.029799) | 0.273383 / 0.323480 (-0.050097) | 0.003984 / 0.007986 (-0.004002) | 0.002791 / 0.004328 (-0.001537) | 0.049076 / 0.004250 (0.044826) | 0.037975 / 0.037052 (0.000922) | 0.253709 / 0.258489 (-0.004780) | 0.281730 / 0.293841 (-0.012111) | 0.028060 / 0.128546 (-0.100486) | 0.010808 / 0.075646 (-0.064838) | 0.206663 / 0.419271 (-0.212609) | 0.035989 / 0.043533 (-0.007544) | 0.252635 / 0.255139 (-0.002504) | 0.280042 / 0.283200 (-0.003158) | 0.016982 / 0.141683 (-0.124700) | 1.098679 / 1.452155 (-0.353475) | 1.157051 / 1.492716 (-0.335666) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.098238 / 0.018006 (0.080232) | 0.311990 / 0.000490 (0.311501) | 0.000229 / 0.000200 (0.000029) | 0.000052 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018270 / 0.037411 (-0.019141) | 0.062711 / 0.014526 (0.048186) | 0.074381 / 0.176557 (-0.102175) | 0.119946 / 0.737135 (-0.617189) | 0.075013 / 0.296338 (-0.221325) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.282106 / 0.215209 (0.066897) | 2.752653 / 2.077655 (0.674999) | 1.488771 / 1.504120 (-0.015349) | 1.372552 / 1.541195 (-0.168643) | 1.390270 / 1.468490 (-0.078220) | 0.558928 / 4.584777 (-4.025849) | 2.411821 / 3.745712 (-1.333891) | 2.771441 / 5.269862 (-2.498421) | 1.747507 / 4.565676 (-2.818169) | 0.061360 / 0.424275 (-0.362915) | 0.004956 / 0.007607 (-0.002652) | 0.332330 / 0.226044 (0.106286) | 3.301405 / 2.268929 (1.032476) | 1.786726 / 55.444624 (-53.657899) | 1.529974 / 6.876477 (-5.346502) | 1.538412 / 2.142072 (-0.603660) | 0.637590 / 4.805227 (-4.167637) | 0.117215 / 6.500664 (-6.383449) | 0.042186 / 0.075469 (-0.033283) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.945574 / 1.841788 (-0.896213) | 11.616152 / 8.074308 (3.541844) | 10.365114 / 10.191392 (0.173722) | 0.130358 / 0.680424 (-0.550066) | 0.013587 / 0.534201 (-0.520614) | 0.306024 / 0.579283 (-0.273259) | 0.270577 / 0.434364 (-0.163787) | 0.340768 / 0.540337 (-0.199569) | 0.460841 / 1.386936 (-0.926095) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005254 / 0.011353 (-0.006099) | 0.003137 / 0.011008 (-0.007871) | 0.048302 / 0.038508 (0.009794) | 0.051952 / 0.023109 (0.028843) | 0.269078 / 0.275898 (-0.006820) | 0.292044 / 0.323480 (-0.031436) | 0.003985 / 0.007986 (-0.004000) | 0.002597 / 0.004328 (-0.001732) | 0.049998 / 0.004250 (0.045747) | 0.040227 / 0.037052 (0.003174) | 0.274714 / 0.258489 (0.016225) | 0.298160 / 0.293841 (0.004319) | 0.028857 / 0.128546 (-0.099690) | 0.010545 / 0.075646 (-0.065101) | 0.057234 / 0.419271 (-0.362038) | 0.032515 / 0.043533 (-0.011018) | 0.271526 / 0.255139 (0.016387) | 0.288556 / 0.283200 (0.005356) | 0.018155 / 0.141683 (-0.123527) | 1.201906 / 1.452155 (-0.250248) | 1.220068 / 1.492716 (-0.272648) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.100098 / 0.018006 (0.082092) | 0.311081 / 0.000490 (0.310591) | 0.000231 / 0.000200 (0.000032) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022349 / 0.037411 (-0.015062) | 0.069698 / 0.014526 (0.055172) | 0.081334 / 0.176557 (-0.095222) | 0.120847 / 0.737135 (-0.616289) | 0.082091 / 0.296338 (-0.214248) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.293810 / 0.215209 (0.078601) | 2.844191 / 2.077655 (0.766536) | 1.594494 / 1.504120 (0.090374) | 1.486531 / 1.541195 (-0.054664) | 1.506307 / 1.468490 (0.037817) | 0.560247 / 4.584777 (-4.024530) | 2.478309 / 3.745712 (-1.267403) | 2.759024 / 5.269862 (-2.510837) | 1.733063 / 4.565676 (-2.832613) | 0.061838 / 0.424275 (-0.362438) | 0.004869 / 0.007607 (-0.002738) | 0.347267 / 0.226044 (0.121222) | 3.407737 / 2.268929 (1.138808) | 1.944420 / 55.444624 (-53.500204) | 1.660060 / 6.876477 (-5.216417) | 1.704219 / 2.142072 (-0.437854) | 0.646969 / 4.805227 (-4.158258) | 0.115750 / 6.500664 (-6.384914) | 0.041614 / 0.075469 (-0.033855) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.972537 / 1.841788 (-0.869251) | 12.013530 / 8.074308 (3.939222) | 10.650215 / 10.191392 (0.458823) | 0.132877 / 0.680424 (-0.547547) | 0.016828 / 0.534201 (-0.517372) | 0.288321 / 0.579283 (-0.290962) | 0.284203 / 0.434364 (-0.150161) | 0.324016 / 0.540337 (-0.216321) | 0.575403 / 1.386936 (-0.811533) |

</details>
</details>

![](https://cml.dev/watermark.png#17ec1a7a610adba3db44f316a930b979872d4ef7 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005925 / 0.011353 (-0.005427) | 0.005138 / 0.011008 (-0.005870) | 0.069865 / 0.038508 (0.031356) | 0.067181 / 0.023109 (0.044072) | 0.309642 / 0.275898 (0.033743) | 0.302919 / 0.323480 (-0.020561) | 0.003365 / 0.007986 (-0.004620) | 0.003148 / 0.004328 (-0.001180) | 0.054102 / 0.004250 (0.049852) | 0.044196 / 0.037052 (0.007143) | 0.306882 / 0.258489 (0.048393) | 0.315153 / 0.293841 (0.021313) | 0.030458 / 0.128546 (-0.098089) | 0.011773 / 0.075646 (-0.063874) | 0.235075 / 0.419271 (-0.184196) | 0.040840 / 0.043533 (-0.002693) | 0.279897 / 0.255139 (0.024758) | 0.316334 / 0.283200 (0.033135) | 0.020128 / 0.141683 (-0.121555) | 1.237327 / 1.452155 (-0.214828) | 1.290386 / 1.492716 (-0.202331) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.118540 / 0.018006 (0.100534) | 0.363282 / 0.000490 (0.362792) | 0.000266 / 0.000200 (0.000066) | 0.000058 / 0.000054 (0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021435 / 0.037411 (-0.015977) | 0.068124 / 0.014526 (0.053598) | 0.082747 / 0.176557 (-0.093809) | 0.137179 / 0.737135 (-0.599956) | 0.084815 / 0.296338 (-0.211523) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.307836 / 0.215209 (0.092626) | 2.983444 / 2.077655 (0.905790) | 1.616430 / 1.504120 (0.112310) | 1.466843 / 1.541195 (-0.074351) | 1.512440 / 1.468490 (0.043950) | 0.652311 / 4.584777 (-3.932466) | 2.676420 / 3.745712 (-1.069292) | 3.265747 / 5.269862 (-2.004115) | 2.028586 / 4.565676 (-2.537090) | 0.071997 / 0.424275 (-0.352278) | 0.007068 / 0.007607 (-0.000539) | 0.367199 / 0.226044 (0.141155) | 3.617970 / 2.268929 (1.349042) | 1.991345 / 55.444624 (-53.453280) | 1.670015 / 6.876477 (-5.206462) | 1.720515 / 2.142072 (-0.421557) | 0.724649 / 4.805227 (-4.080579) | 0.134888 / 6.500664 (-6.365776) | 0.048325 / 0.075469 (-0.027144) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.051058 / 1.841788 (-0.790730) | 13.772809 / 8.074308 (5.698501) | 11.813879 / 10.191392 (1.622487) | 0.160065 / 0.680424 (-0.520359) | 0.016256 / 0.534201 (-0.517945) | 0.320393 / 0.579283 (-0.258890) | 0.314462 / 0.434364 (-0.119901) | 0.371911 / 0.540337 (-0.168427) | 0.506864 / 1.386936 (-0.880072) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005857 / 0.011353 (-0.005496) | 0.004077 / 0.011008 (-0.006931) | 0.056033 / 0.038508 (0.017525) | 0.067622 / 0.023109 (0.044513) | 0.298956 / 0.275898 (0.023058) | 0.323484 / 0.323480 (0.000004) | 0.004825 / 0.007986 (-0.003160) | 0.003120 / 0.004328 (-0.001208) | 0.055227 / 0.004250 (0.050976) | 0.048439 / 0.037052 (0.011387) | 0.303207 / 0.258489 (0.044718) | 0.329478 / 0.293841 (0.035637) | 0.032516 / 0.128546 (-0.096031) | 0.012260 / 0.075646 (-0.063386) | 0.065037 / 0.419271 (-0.354234) | 0.038799 / 0.043533 (-0.004734) | 0.299102 / 0.255139 (0.043963) | 0.318248 / 0.283200 (0.035048) | 0.020190 / 0.141683 (-0.121493) | 1.263479 / 1.452155 (-0.188676) | 1.329788 / 1.492716 (-0.162928) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.119801 / 0.018006 (0.101794) | 0.359618 / 0.000490 (0.359129) | 0.000260 / 0.000200 (0.000060) | 0.000058 / 0.000054 (0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026876 / 0.037411 (-0.010535) | 0.080637 / 0.014526 (0.066111) | 0.092260 / 0.176557 (-0.084297) | 0.137260 / 0.737135 (-0.599875) | 0.093309 / 0.296338 (-0.203029) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.329327 / 0.215209 (0.114118) | 3.193014 / 2.077655 (1.115359) | 1.755838 / 1.504120 (0.251718) | 1.612279 / 1.541195 (0.071084) | 1.631958 / 1.468490 (0.163468) | 0.630886 / 4.584777 (-3.953891) | 2.739731 / 3.745712 (-1.005981) | 3.186745 / 5.269862 (-2.083117) | 1.987125 / 4.565676 (-2.578552) | 0.070694 / 0.424275 (-0.353581) | 0.006461 / 0.007607 (-0.001146) | 0.386367 / 0.226044 (0.160323) | 3.815837 / 2.268929 (1.546908) | 2.155904 / 55.444624 (-53.288720) | 1.832575 / 6.876477 (-5.043902) | 1.842097 / 2.142072 (-0.299975) | 0.716394 / 4.805227 (-4.088833) | 0.130796 / 6.500664 (-6.369869) | 0.045674 / 0.075469 (-0.029795) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.109117 / 1.841788 (-0.732671) | 14.116582 / 8.074308 (6.042274) | 11.926356 / 10.191392 (1.734964) | 0.150543 / 0.680424 (-0.529881) | 0.017426 / 0.534201 (-0.516775) | 0.323058 / 0.579283 (-0.256225) | 0.330228 / 0.434364 (-0.104136) | 0.372533 / 0.540337 (-0.167804) | 0.661348 / 1.386936 (-0.725588) |

</details>
</details>

![](https://cml.dev/watermark.png#04ffd22a30ecc7545234559edd9d23c85c6d84d9 "CML watermark")


Thanks for the review, I took your comments into account !

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005477 / 0.011353 (-0.005876) | 0.003509 / 0.011008 (-0.007499) | 0.062884 / 0.038508 (0.024376) | 0.051042 / 0.023109 (0.027933) | 0.285180 / 0.275898 (0.009282) | 0.315353 / 0.323480 (-0.008127) | 0.002943 / 0.007986 (-0.005043) | 0.003286 / 0.004328 (-0.001042) | 0.048885 / 0.004250 (0.044635) | 0.038591 / 0.037052 (0.001539) | 0.288527 / 0.258489 (0.030038) | 0.316102 / 0.293841 (0.022261) | 0.028252 / 0.128546 (-0.100295) | 0.010622 / 0.075646 (-0.065024) | 0.205573 / 0.419271 (-0.213699) | 0.035764 / 0.043533 (-0.007769) | 0.285729 / 0.255139 (0.030590) | 0.304578 / 0.283200 (0.021378) | 0.019862 / 0.141683 (-0.121821) | 1.102866 / 1.452155 (-0.349288) | 1.175161 / 1.492716 (-0.317555) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095253 / 0.018006 (0.077246) | 0.302290 / 0.000490 (0.301800) | 0.000243 / 0.000200 (0.000043) | 0.000061 / 0.000054 (0.000007) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018680 / 0.037411 (-0.018731) | 0.060375 / 0.014526 (0.045849) | 0.074033 / 0.176557 (-0.102524) | 0.120290 / 0.737135 (-0.616845) | 0.075350 / 0.296338 (-0.220989) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.277617 / 0.215209 (0.062408) | 2.718201 / 2.077655 (0.640546) | 1.462952 / 1.504120 (-0.041168) | 1.339199 / 1.541195 (-0.201996) | 1.375805 / 1.468490 (-0.092685) | 0.559956 / 4.584777 (-4.024821) | 2.373865 / 3.745712 (-1.371847) | 2.795732 / 5.269862 (-2.474129) | 1.755490 / 4.565676 (-2.810186) | 0.062002 / 0.424275 (-0.362273) | 0.004935 / 0.007607 (-0.002672) | 0.334786 / 0.226044 (0.108741) | 3.237499 / 2.268929 (0.968571) | 1.787561 / 55.444624 (-53.657064) | 1.513300 / 6.876477 (-5.363176) | 1.549797 / 2.142072 (-0.592275) | 0.643587 / 4.805227 (-4.161640) | 0.117275 / 6.500664 (-6.383389) | 0.042184 / 0.075469 (-0.033285) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.933366 / 1.841788 (-0.908421) | 11.792282 / 8.074308 (3.717973) | 10.466608 / 10.191392 (0.275216) | 0.142148 / 0.680424 (-0.538275) | 0.014084 / 0.534201 (-0.520117) | 0.287233 / 0.579283 (-0.292050) | 0.266022 / 0.434364 (-0.168342) | 0.326854 / 0.540337 (-0.213483) | 0.451348 / 1.386936 (-0.935588) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005384 / 0.011353 (-0.005969) | 0.003562 / 0.011008 (-0.007446) | 0.049014 / 0.038508 (0.010506) | 0.057480 / 0.023109 (0.034371) | 0.274456 / 0.275898 (-0.001442) | 0.298387 / 0.323480 (-0.025093) | 0.003909 / 0.007986 (-0.004076) | 0.002646 / 0.004328 (-0.001683) | 0.048374 / 0.004250 (0.044124) | 0.040907 / 0.037052 (0.003854) | 0.278267 / 0.258489 (0.019778) | 0.299862 / 0.293841 (0.006021) | 0.029108 / 0.128546 (-0.099439) | 0.010752 / 0.075646 (-0.064894) | 0.057523 / 0.419271 (-0.361749) | 0.032692 / 0.043533 (-0.010841) | 0.276288 / 0.255139 (0.021149) | 0.291572 / 0.283200 (0.008372) | 0.017818 / 0.141683 (-0.123865) | 1.129517 / 1.452155 (-0.322638) | 1.186630 / 1.492716 (-0.306086) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093405 / 0.018006 (0.075399) | 0.301254 / 0.000490 (0.300764) | 0.000225 / 0.000200 (0.000025) | 0.000054 / 0.000054 (-0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021793 / 0.037411 (-0.015618) | 0.069033 / 0.014526 (0.054508) | 0.083502 / 0.176557 (-0.093055) | 0.122149 / 0.737135 (-0.614986) | 0.083801 / 0.296338 (-0.212537) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.299149 / 0.215209 (0.083940) | 2.936550 / 2.077655 (0.858895) | 1.595766 / 1.504120 (0.091647) | 1.487117 / 1.541195 (-0.054078) | 1.494606 / 1.468490 (0.026116) | 0.569346 / 4.584777 (-4.015431) | 2.445642 / 3.745712 (-1.300070) | 2.805696 / 5.269862 (-2.464165) | 1.743796 / 4.565676 (-2.821881) | 0.062695 / 0.424275 (-0.361580) | 0.004885 / 0.007607 (-0.002723) | 0.354186 / 0.226044 (0.128142) | 3.487926 / 2.268929 (1.218997) | 1.965703 / 55.444624 (-53.478922) | 1.682284 / 6.876477 (-5.194193) | 1.705586 / 2.142072 (-0.436487) | 0.655099 / 4.805227 (-4.150128) | 0.116441 / 6.500664 (-6.384223) | 0.040851 / 0.075469 (-0.034618) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.967361 / 1.841788 (-0.874427) | 12.037718 / 8.074308 (3.963409) | 10.599761 / 10.191392 (0.408369) | 0.143127 / 0.680424 (-0.537297) | 0.015063 / 0.534201 (-0.519138) | 0.286894 / 0.579283 (-0.292389) | 0.301505 / 0.434364 (-0.132859) | 0.324339 / 0.540337 (-0.215999) | 0.591782 / 1.386936 (-0.795154) |

</details>
</details>

![](https://cml.dev/watermark.png#b96ff08d4aa6dbafc8a10a9d03dfabe236378bcd "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005337 / 0.011353 (-0.006015) | 0.004074 / 0.011008 (-0.006934) | 0.062653 / 0.038508 (0.024145) | 0.054295 / 0.023109 (0.031186) | 0.248284 / 0.275898 (-0.027614) | 0.271604 / 0.323480 (-0.051876) | 0.003931 / 0.007986 (-0.004055) | 0.002907 / 0.004328 (-0.001422) | 0.047991 / 0.004250 (0.043740) | 0.042842 / 0.037052 (0.005790) | 0.253648 / 0.258489 (-0.004841) | 0.282546 / 0.293841 (-0.011295) | 0.028005 / 0.128546 (-0.100541) | 0.010734 / 0.075646 (-0.064912) | 0.210023 / 0.419271 (-0.209248) | 0.035940 / 0.043533 (-0.007592) | 0.250766 / 0.255139 (-0.004373) | 0.267644 / 0.283200 (-0.015556) | 0.020451 / 0.141683 (-0.121232) | 1.114972 / 1.452155 (-0.337183) | 1.159823 / 1.492716 (-0.332893) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095527 / 0.018006 (0.077521) | 0.303321 / 0.000490 (0.302831) | 0.000216 / 0.000200 (0.000016) | 0.000048 / 0.000054 (-0.000006) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018725 / 0.037411 (-0.018686) | 0.062537 / 0.014526 (0.048011) | 0.073091 / 0.176557 (-0.103466) | 0.119570 / 0.737135 (-0.617565) | 0.074863 / 0.296338 (-0.221476) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.284936 / 0.215209 (0.069727) | 2.802498 / 2.077655 (0.724843) | 1.493316 / 1.504120 (-0.010804) | 1.372319 / 1.541195 (-0.168875) | 1.403657 / 1.468490 (-0.064833) | 0.569303 / 4.584777 (-4.015474) | 2.402498 / 3.745712 (-1.343214) | 2.834778 / 5.269862 (-2.435084) | 1.791312 / 4.565676 (-2.774365) | 0.062526 / 0.424275 (-0.361749) | 0.004947 / 0.007607 (-0.002660) | 0.345141 / 0.226044 (0.119097) | 3.371863 / 2.268929 (1.102934) | 1.846023 / 55.444624 (-53.598602) | 1.596368 / 6.876477 (-5.280109) | 1.615902 / 2.142072 (-0.526170) | 0.644333 / 4.805227 (-4.160894) | 0.119460 / 6.500664 (-6.381204) | 0.049122 / 0.075469 (-0.026347) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.951839 / 1.841788 (-0.889948) | 11.677074 / 8.074308 (3.602766) | 10.562586 / 10.191392 (0.371194) | 0.143633 / 0.680424 (-0.536791) | 0.014157 / 0.534201 (-0.520044) | 0.289141 / 0.579283 (-0.290142) | 0.264719 / 0.434364 (-0.169645) | 0.327862 / 0.540337 (-0.212476) | 0.451215 / 1.386936 (-0.935721) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005343 / 0.011353 (-0.006010) | 0.003522 / 0.011008 (-0.007486) | 0.049354 / 0.038508 (0.010846) | 0.051441 / 0.023109 (0.028332) | 0.259350 / 0.275898 (-0.016548) | 0.288946 / 0.323480 (-0.034534) | 0.004052 / 0.007986 (-0.003934) | 0.002690 / 0.004328 (-0.001639) | 0.049996 / 0.004250 (0.045746) | 0.040224 / 0.037052 (0.003171) | 0.264588 / 0.258489 (0.006099) | 0.296474 / 0.293841 (0.002633) | 0.028868 / 0.128546 (-0.099679) | 0.010917 / 0.075646 (-0.064730) | 0.057866 / 0.419271 (-0.361405) | 0.032610 / 0.043533 (-0.010923) | 0.260657 / 0.255139 (0.005518) | 0.276947 / 0.283200 (-0.006253) | 0.018877 / 0.141683 (-0.122806) | 1.126205 / 1.452155 (-0.325949) | 1.206173 / 1.492716 (-0.286543) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094464 / 0.018006 (0.076458) | 0.304473 / 0.000490 (0.303984) | 0.000231 / 0.000200 (0.000031) | 0.000053 / 0.000054 (-0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021472 / 0.037411 (-0.015939) | 0.070864 / 0.014526 (0.056338) | 0.086607 / 0.176557 (-0.089950) | 0.120679 / 0.737135 (-0.616456) | 0.084271 / 0.296338 (-0.212068) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.296448 / 0.215209 (0.081239) | 2.893996 / 2.077655 (0.816341) | 1.573409 / 1.504120 (0.069289) | 1.438799 / 1.541195 (-0.102396) | 1.461241 / 1.468490 (-0.007249) | 0.566737 / 4.584777 (-4.018040) | 2.425709 / 3.745712 (-1.320003) | 2.826764 / 5.269862 (-2.443098) | 1.785330 / 4.565676 (-2.780347) | 0.063721 / 0.424275 (-0.360554) | 0.005158 / 0.007607 (-0.002449) | 0.354961 / 0.226044 (0.128916) | 3.457499 / 2.268929 (1.188570) | 1.931374 / 55.444624 (-53.513251) | 1.646515 / 6.876477 (-5.229962) | 1.629891 / 2.142072 (-0.512182) | 0.648922 / 4.805227 (-4.156305) | 0.114953 / 6.500664 (-6.385711) | 0.040997 / 0.075469 (-0.034472) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.951049 / 1.841788 (-0.890739) | 12.258298 / 8.074308 (4.183990) | 10.663309 / 10.191392 (0.471917) | 0.142933 / 0.680424 (-0.537491) | 0.015927 / 0.534201 (-0.518273) | 0.286914 / 0.579283 (-0.292369) | 0.286600 / 0.434364 (-0.147764) | 0.324464 / 0.540337 (-0.215874) | 0.575075 / 1.386936 (-0.811861) |

</details>
</details>

![](https://cml.dev/watermark.png#ed47b9d5e9c6aa03a0aa07d8abfd3fa8241da353 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005298 / 0.011353 (-0.006055) | 0.003645 / 0.011008 (-0.007363) | 0.061629 / 0.038508 (0.023121) | 0.052322 / 0.023109 (0.029212) | 0.242579 / 0.275898 (-0.033319) | 0.263525 / 0.323480 (-0.059955) | 0.002794 / 0.007986 (-0.005192) | 0.002152 / 0.004328 (-0.002177) | 0.048301 / 0.004250 (0.044050) | 0.038177 / 0.037052 (0.001125) | 0.247724 / 0.258489 (-0.010765) | 0.274455 / 0.293841 (-0.019386) | 0.026992 / 0.128546 (-0.101555) | 0.010110 / 0.075646 (-0.065536) | 0.205662 / 0.419271 (-0.213609) | 0.034901 / 0.043533 (-0.008632) | 0.241920 / 0.255139 (-0.013219) | 0.262048 / 0.283200 (-0.021152) | 0.019111 / 0.141683 (-0.122572) | 1.127600 / 1.452155 (-0.324555) | 1.193931 / 1.492716 (-0.298786) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090321 / 0.018006 (0.072315) | 0.299046 / 0.000490 (0.298556) | 0.000197 / 0.000200 (-0.000003) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018278 / 0.037411 (-0.019133) | 0.060114 / 0.014526 (0.045588) | 0.073602 / 0.176557 (-0.102954) | 0.119676 / 0.737135 (-0.617459) | 0.074786 / 0.296338 (-0.221552) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.280385 / 0.215209 (0.065176) | 2.764259 / 2.077655 (0.686604) | 1.501027 / 1.504120 (-0.003093) | 1.376900 / 1.541195 (-0.164295) | 1.390587 / 1.468490 (-0.077903) | 0.555180 / 4.584777 (-4.029597) | 2.354307 / 3.745712 (-1.391405) | 2.755862 / 5.269862 (-2.514000) | 1.714771 / 4.565676 (-2.850906) | 0.062507 / 0.424275 (-0.361768) | 0.004974 / 0.007607 (-0.002633) | 0.333900 / 0.226044 (0.107856) | 3.266922 / 2.268929 (0.997994) | 1.805401 / 55.444624 (-53.639223) | 1.526970 / 6.876477 (-5.349507) | 1.539425 / 2.142072 (-0.602647) | 0.629364 / 4.805227 (-4.175863) | 0.114929 / 6.500664 (-6.385735) | 0.041258 / 0.075469 (-0.034211) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.968601 / 1.841788 (-0.873187) | 11.260937 / 8.074308 (3.186629) | 10.393839 / 10.191392 (0.202447) | 0.127988 / 0.680424 (-0.552436) | 0.014564 / 0.534201 (-0.519637) | 0.286560 / 0.579283 (-0.292723) | 0.260493 / 0.434364 (-0.173871) | 0.330949 / 0.540337 (-0.209388) | 0.435798 / 1.386936 (-0.951138) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005232 / 0.011353 (-0.006121) | 0.003030 / 0.011008 (-0.007978) | 0.048513 / 0.038508 (0.010005) | 0.049501 / 0.023109 (0.026392) | 0.270545 / 0.275898 (-0.005353) | 0.289128 / 0.323480 (-0.034352) | 0.003925 / 0.007986 (-0.004061) | 0.002568 / 0.004328 (-0.001761) | 0.047692 / 0.004250 (0.043442) | 0.039854 / 0.037052 (0.002802) | 0.272654 / 0.258489 (0.014165) | 0.296275 / 0.293841 (0.002434) | 0.029027 / 0.128546 (-0.099519) | 0.010335 / 0.075646 (-0.065311) | 0.056726 / 0.419271 (-0.362546) | 0.033257 / 0.043533 (-0.010275) | 0.272672 / 0.255139 (0.017533) | 0.286298 / 0.283200 (0.003098) | 0.017877 / 0.141683 (-0.123806) | 1.150322 / 1.452155 (-0.301833) | 1.221031 / 1.492716 (-0.271685) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.102838 / 0.018006 (0.084832) | 0.298810 / 0.000490 (0.298320) | 0.000207 / 0.000200 (0.000007) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021232 / 0.037411 (-0.016180) | 0.067949 / 0.014526 (0.053423) | 0.116487 / 0.176557 (-0.060070) | 0.124035 / 0.737135 (-0.613100) | 0.081075 / 0.296338 (-0.215263) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.289098 / 0.215209 (0.073889) | 2.844476 / 2.077655 (0.766821) | 1.609576 / 1.504120 (0.105456) | 1.480453 / 1.541195 (-0.060742) | 1.489672 / 1.468490 (0.021182) | 0.589661 / 4.584777 (-3.995116) | 2.453804 / 3.745712 (-1.291908) | 2.722381 / 5.269862 (-2.547480) | 1.720251 / 4.565676 (-2.845425) | 0.066085 / 0.424275 (-0.358190) | 0.004943 / 0.007607 (-0.002664) | 0.355149 / 0.226044 (0.129104) | 3.444323 / 2.268929 (1.175395) | 1.971157 / 55.444624 (-53.473467) | 1.683029 / 6.876477 (-5.193448) | 1.672798 / 2.142072 (-0.469274) | 0.644812 / 4.805227 (-4.160416) | 0.115098 / 6.500664 (-6.385566) | 0.039883 / 0.075469 (-0.035586) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.960454 / 1.841788 (-0.881334) | 11.604732 / 8.074308 (3.530424) | 10.405481 / 10.191392 (0.214089) | 0.129146 / 0.680424 (-0.551278) | 0.014945 / 0.534201 (-0.519256) | 0.286239 / 0.579283 (-0.293044) | 0.281041 / 0.434364 (-0.153323) | 0.320448 / 0.540337 (-0.219890) | 0.554304 / 1.386936 (-0.832632) |

</details>
</details>

![](https://cml.dev/watermark.png#b2cfb7859b029654829c4dfee230812ddab1f104 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005510 / 0.011353 (-0.005843) | 0.003575 / 0.011008 (-0.007433) | 0.062232 / 0.038508 (0.023724) | 0.051115 / 0.023109 (0.028006) | 0.250709 / 0.275898 (-0.025189) | 0.274837 / 0.323480 (-0.048642) | 0.002972 / 0.007986 (-0.005014) | 0.002708 / 0.004328 (-0.001621) | 0.048088 / 0.004250 (0.043838) | 0.038588 / 0.037052 (0.001535) | 0.252550 / 0.258489 (-0.005939) | 0.285238 / 0.293841 (-0.008603) | 0.027867 / 0.128546 (-0.100679) | 0.011000 / 0.075646 (-0.064646) | 0.206918 / 0.419271 (-0.212354) | 0.035711 / 0.043533 (-0.007822) | 0.255306 / 0.255139 (0.000167) | 0.298636 / 0.283200 (0.015436) | 0.018222 / 0.141683 (-0.123461) | 1.122276 / 1.452155 (-0.329879) | 1.196471 / 1.492716 (-0.296245) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092072 / 0.018006 (0.074066) | 0.301469 / 0.000490 (0.300979) | 0.000225 / 0.000200 (0.000025) | 0.000050 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018672 / 0.037411 (-0.018739) | 0.060235 / 0.014526 (0.045709) | 0.074036 / 0.176557 (-0.102521) | 0.119578 / 0.737135 (-0.617557) | 0.073605 / 0.296338 (-0.222734) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.286474 / 0.215209 (0.071264) | 2.779427 / 2.077655 (0.701772) | 1.478746 / 1.504120 (-0.025373) | 1.362692 / 1.541195 (-0.178503) | 1.388194 / 1.468490 (-0.080296) | 0.560707 / 4.584777 (-4.024070) | 2.352846 / 3.745712 (-1.392866) | 2.784400 / 5.269862 (-2.485461) | 1.775642 / 4.565676 (-2.790035) | 0.062324 / 0.424275 (-0.361951) | 0.004938 / 0.007607 (-0.002669) | 0.334149 / 0.226044 (0.108105) | 3.319446 / 2.268929 (1.050517) | 1.810369 / 55.444624 (-53.634255) | 1.559462 / 6.876477 (-5.317014) | 1.611199 / 2.142072 (-0.530873) | 0.655984 / 4.805227 (-4.149244) | 0.118508 / 6.500664 (-6.382156) | 0.043661 / 0.075469 (-0.031808) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.935046 / 1.841788 (-0.906742) | 11.413501 / 8.074308 (3.339192) | 10.392314 / 10.191392 (0.200922) | 0.131507 / 0.680424 (-0.548917) | 0.014827 / 0.534201 (-0.519374) | 0.289069 / 0.579283 (-0.290214) | 0.268288 / 0.434364 (-0.166076) | 0.326843 / 0.540337 (-0.213495) | 0.441283 / 1.386936 (-0.945653) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005375 / 0.011353 (-0.005978) | 0.003549 / 0.011008 (-0.007459) | 0.048996 / 0.038508 (0.010488) | 0.051408 / 0.023109 (0.028298) | 0.272265 / 0.275898 (-0.003633) | 0.293228 / 0.323480 (-0.030252) | 0.004147 / 0.007986 (-0.003839) | 0.002673 / 0.004328 (-0.001655) | 0.048116 / 0.004250 (0.043865) | 0.039926 / 0.037052 (0.002874) | 0.276987 / 0.258489 (0.018498) | 0.302955 / 0.293841 (0.009115) | 0.029488 / 0.128546 (-0.099058) | 0.010797 / 0.075646 (-0.064849) | 0.057552 / 0.419271 (-0.361720) | 0.032827 / 0.043533 (-0.010706) | 0.270888 / 0.255139 (0.015749) | 0.289136 / 0.283200 (0.005937) | 0.018815 / 0.141683 (-0.122868) | 1.148624 / 1.452155 (-0.303530) | 1.191184 / 1.492716 (-0.301532) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091712 / 0.018006 (0.073706) | 0.311198 / 0.000490 (0.310708) | 0.000226 / 0.000200 (0.000026) | 0.000049 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022097 / 0.037411 (-0.015314) | 0.070641 / 0.014526 (0.056116) | 0.080084 / 0.176557 (-0.096472) | 0.118998 / 0.737135 (-0.618137) | 0.081827 / 0.296338 (-0.214512) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.298599 / 0.215209 (0.083390) | 2.884759 / 2.077655 (0.807105) | 1.630794 / 1.504120 (0.126674) | 1.454309 / 1.541195 (-0.086886) | 1.466795 / 1.468490 (-0.001695) | 0.565405 / 4.584777 (-4.019372) | 2.460883 / 3.745712 (-1.284829) | 2.764193 / 5.269862 (-2.505668) | 1.734270 / 4.565676 (-2.831407) | 0.063408 / 0.424275 (-0.360867) | 0.004887 / 0.007607 (-0.002720) | 0.347762 / 0.226044 (0.121717) | 3.458385 / 2.268929 (1.189457) | 1.965434 / 55.444624 (-53.479190) | 1.671047 / 6.876477 (-5.205430) | 1.665642 / 2.142072 (-0.476430) | 0.640665 / 4.805227 (-4.164562) | 0.116025 / 6.500664 (-6.384639) | 0.040147 / 0.075469 (-0.035322) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.982194 / 1.841788 (-0.859593) | 11.983487 / 8.074308 (3.909179) | 10.660605 / 10.191392 (0.469213) | 0.140647 / 0.680424 (-0.539777) | 0.015870 / 0.534201 (-0.518331) | 0.287032 / 0.579283 (-0.292251) | 0.276629 / 0.434364 (-0.157735) | 0.331171 / 0.540337 (-0.209166) | 0.575346 / 1.386936 (-0.811590) |

</details>
</details>

![](https://cml.dev/watermark.png#56433c2f6a42d5fcc5acb46c6275911c29afc371 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005014 / 0.011353 (-0.006339) | 0.003434 / 0.011008 (-0.007574) | 0.063283 / 0.038508 (0.024775) | 0.048068 / 0.023109 (0.024959) | 0.239521 / 0.275898 (-0.036377) | 0.265294 / 0.323480 (-0.058186) | 0.003790 / 0.007986 (-0.004196) | 0.002577 / 0.004328 (-0.001751) | 0.048618 / 0.004250 (0.044368) | 0.037427 / 0.037052 (0.000375) | 0.245263 / 0.258489 (-0.013226) | 0.276618 / 0.293841 (-0.017223) | 0.026615 / 0.128546 (-0.101931) | 0.010378 / 0.075646 (-0.065268) | 0.205670 / 0.419271 (-0.213601) | 0.035076 / 0.043533 (-0.008457) | 0.245062 / 0.255139 (-0.010077) | 0.264584 / 0.283200 (-0.018616) | 0.017760 / 0.141683 (-0.123922) | 1.148061 / 1.452155 (-0.304094) | 1.192762 / 1.492716 (-0.299955) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090870 / 0.018006 (0.072864) | 0.305458 / 0.000490 (0.304968) | 0.000207 / 0.000200 (0.000007) | 0.000052 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018597 / 0.037411 (-0.018814) | 0.060349 / 0.014526 (0.045823) | 0.074854 / 0.176557 (-0.101702) | 0.123243 / 0.737135 (-0.613892) | 0.075843 / 0.296338 (-0.220496) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.275855 / 0.215209 (0.060645) | 2.723965 / 2.077655 (0.646311) | 1.436010 / 1.504120 (-0.068110) | 1.323495 / 1.541195 (-0.217700) | 1.356234 / 1.468490 (-0.112256) | 0.564388 / 4.584777 (-4.020389) | 2.390180 / 3.745712 (-1.355532) | 2.782863 / 5.269862 (-2.486998) | 1.765048 / 4.565676 (-2.800628) | 0.062680 / 0.424275 (-0.361595) | 0.004929 / 0.007607 (-0.002678) | 0.337578 / 0.226044 (0.111533) | 3.316780 / 2.268929 (1.047851) | 1.803829 / 55.444624 (-53.640795) | 1.524585 / 6.876477 (-5.351891) | 1.549695 / 2.142072 (-0.592377) | 0.638053 / 4.805227 (-4.167174) | 0.116983 / 6.500664 (-6.383681) | 0.042251 / 0.075469 (-0.033218) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.946978 / 1.841788 (-0.894810) | 11.809483 / 8.074308 (3.735175) | 10.459974 / 10.191392 (0.268582) | 0.130015 / 0.680424 (-0.550409) | 0.013843 / 0.534201 (-0.520358) | 0.286972 / 0.579283 (-0.292311) | 0.268904 / 0.434364 (-0.165460) | 0.325591 / 0.540337 (-0.214746) | 0.439233 / 1.386936 (-0.947703) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005804 / 0.011353 (-0.005549) | 0.003431 / 0.011008 (-0.007577) | 0.049041 / 0.038508 (0.010533) | 0.054758 / 0.023109 (0.031649) | 0.262330 / 0.275898 (-0.013568) | 0.288872 / 0.323480 (-0.034608) | 0.004016 / 0.007986 (-0.003970) | 0.002606 / 0.004328 (-0.001722) | 0.047878 / 0.004250 (0.043628) | 0.045066 / 0.037052 (0.008013) | 0.266310 / 0.258489 (0.007820) | 0.290072 / 0.293841 (-0.003768) | 0.028738 / 0.128546 (-0.099809) | 0.010667 / 0.075646 (-0.064979) | 0.057300 / 0.419271 (-0.361972) | 0.032715 / 0.043533 (-0.010818) | 0.264043 / 0.255139 (0.008904) | 0.278652 / 0.283200 (-0.004547) | 0.017873 / 0.141683 (-0.123810) | 1.125981 / 1.452155 (-0.326174) | 1.168548 / 1.492716 (-0.324168) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090997 / 0.018006 (0.072991) | 0.300807 / 0.000490 (0.300317) | 0.000223 / 0.000200 (0.000023) | 0.000043 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021510 / 0.037411 (-0.015901) | 0.068251 / 0.014526 (0.053725) | 0.082073 / 0.176557 (-0.094484) | 0.120071 / 0.737135 (-0.617064) | 0.082245 / 0.296338 (-0.214093) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.290601 / 0.215209 (0.075392) | 2.871855 / 2.077655 (0.794200) | 1.558239 / 1.504120 (0.054119) | 1.447767 / 1.541195 (-0.093427) | 1.446851 / 1.468490 (-0.021639) | 0.573990 / 4.584777 (-4.010787) | 2.439859 / 3.745712 (-1.305853) | 2.795899 / 5.269862 (-2.473963) | 1.746751 / 4.565676 (-2.818926) | 0.062100 / 0.424275 (-0.362175) | 0.004948 / 0.007607 (-0.002659) | 0.344281 / 0.226044 (0.118236) | 3.427499 / 2.268929 (1.158570) | 1.940348 / 55.444624 (-53.504276) | 1.660926 / 6.876477 (-5.215551) | 1.669485 / 2.142072 (-0.472588) | 0.634034 / 4.805227 (-4.171193) | 0.114748 / 6.500664 (-6.385916) | 0.041617 / 0.075469 (-0.033852) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.966411 / 1.841788 (-0.875376) | 12.040753 / 8.074308 (3.966445) | 10.506542 / 10.191392 (0.315150) | 0.129659 / 0.680424 (-0.550764) | 0.015691 / 0.534201 (-0.518510) | 0.286911 / 0.579283 (-0.292372) | 0.273588 / 0.434364 (-0.160776) | 0.333642 / 0.540337 (-0.206695) | 0.568550 / 1.386936 (-0.818386) |

</details>
</details>

![](https://cml.dev/watermark.png#b38ed4705263df92ae06d89baab0932ae10065e0 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005023 / 0.011353 (-0.006330) | 0.003492 / 0.011008 (-0.007516) | 0.062808 / 0.038508 (0.024300) | 0.051649 / 0.023109 (0.028540) | 0.246871 / 0.275898 (-0.029027) | 0.273430 / 0.323480 (-0.050050) | 0.003851 / 0.007986 (-0.004135) | 0.002643 / 0.004328 (-0.001686) | 0.048499 / 0.004250 (0.044248) | 0.037713 / 0.037052 (0.000661) | 0.256431 / 0.258489 (-0.002058) | 0.306956 / 0.293841 (0.013116) | 0.027116 / 0.128546 (-0.101430) | 0.010769 / 0.075646 (-0.064877) | 0.206218 / 0.419271 (-0.213053) | 0.035592 / 0.043533 (-0.007941) | 0.249629 / 0.255139 (-0.005510) | 0.268438 / 0.283200 (-0.014761) | 0.018557 / 0.141683 (-0.123125) | 1.123988 / 1.452155 (-0.328167) | 1.158196 / 1.492716 (-0.334520) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090221 / 0.018006 (0.072215) | 0.300892 / 0.000490 (0.300402) | 0.000209 / 0.000200 (0.000009) | 0.000046 / 0.000054 (-0.000008) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018305 / 0.037411 (-0.019106) | 0.060294 / 0.014526 (0.045769) | 0.073330 / 0.176557 (-0.103227) | 0.119620 / 0.737135 (-0.617515) | 0.074611 / 0.296338 (-0.221727) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.285347 / 0.215209 (0.070138) | 2.795144 / 2.077655 (0.717490) | 1.468321 / 1.504120 (-0.035799) | 1.343848 / 1.541195 (-0.197347) | 1.388998 / 1.468490 (-0.079492) | 0.559609 / 4.584777 (-4.025168) | 2.355056 / 3.745712 (-1.390656) | 2.798763 / 5.269862 (-2.471099) | 1.764371 / 4.565676 (-2.801305) | 0.062563 / 0.424275 (-0.361712) | 0.005101 / 0.007607 (-0.002506) | 0.339205 / 0.226044 (0.113161) | 3.336729 / 2.268929 (1.067800) | 1.801987 / 55.444624 (-53.642637) | 1.526720 / 6.876477 (-5.349757) | 1.539324 / 2.142072 (-0.602749) | 0.635805 / 4.805227 (-4.169422) | 0.138762 / 6.500664 (-6.361902) | 0.042092 / 0.075469 (-0.033377) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.928755 / 1.841788 (-0.913032) | 11.468224 / 8.074308 (3.393916) | 10.784568 / 10.191392 (0.593176) | 0.130332 / 0.680424 (-0.550092) | 0.014203 / 0.534201 (-0.519998) | 0.287125 / 0.579283 (-0.292158) | 0.263921 / 0.434364 (-0.170443) | 0.327824 / 0.540337 (-0.212513) | 0.434679 / 1.386936 (-0.952257) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005194 / 0.011353 (-0.006159) | 0.003411 / 0.011008 (-0.007598) | 0.050122 / 0.038508 (0.011614) | 0.049378 / 0.023109 (0.026269) | 0.272980 / 0.275898 (-0.002918) | 0.298047 / 0.323480 (-0.025433) | 0.003945 / 0.007986 (-0.004041) | 0.002633 / 0.004328 (-0.001696) | 0.048935 / 0.004250 (0.044685) | 0.040157 / 0.037052 (0.003104) | 0.277056 / 0.258489 (0.018567) | 0.299824 / 0.293841 (0.005983) | 0.028997 / 0.128546 (-0.099550) | 0.010868 / 0.075646 (-0.064779) | 0.057895 / 0.419271 (-0.361377) | 0.033522 / 0.043533 (-0.010010) | 0.274912 / 0.255139 (0.019773) | 0.288902 / 0.283200 (0.005702) | 0.018016 / 0.141683 (-0.123667) | 1.116669 / 1.452155 (-0.335485) | 1.175007 / 1.492716 (-0.317710) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090169 / 0.018006 (0.072163) | 0.310577 / 0.000490 (0.310087) | 0.000215 / 0.000200 (0.000015) | 0.000048 / 0.000054 (-0.000006) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.020448 / 0.037411 (-0.016963) | 0.068216 / 0.014526 (0.053690) | 0.081798 / 0.176557 (-0.094759) | 0.119151 / 0.737135 (-0.617985) | 0.085197 / 0.296338 (-0.211142) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.294957 / 0.215209 (0.079748) | 2.874065 / 2.077655 (0.796410) | 1.590963 / 1.504120 (0.086843) | 1.459596 / 1.541195 (-0.081599) | 1.467931 / 1.468490 (-0.000559) | 0.562832 / 4.584777 (-4.021944) | 2.426384 / 3.745712 (-1.319328) | 2.767749 / 5.269862 (-2.502112) | 1.746702 / 4.565676 (-2.818975) | 0.063353 / 0.424275 (-0.360922) | 0.005073 / 0.007607 (-0.002534) | 0.348258 / 0.226044 (0.122213) | 3.390351 / 2.268929 (1.121423) | 1.950092 / 55.444624 (-53.494532) | 1.671227 / 6.876477 (-5.205250) | 1.683349 / 2.142072 (-0.458723) | 0.637613 / 4.805227 (-4.167614) | 0.115172 / 6.500664 (-6.385492) | 0.040202 / 0.075469 (-0.035267) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.963085 / 1.841788 (-0.878702) | 11.895384 / 8.074308 (3.821076) | 10.609906 / 10.191392 (0.418513) | 0.130865 / 0.680424 (-0.549559) | 0.016020 / 0.534201 (-0.518181) | 0.287540 / 0.579283 (-0.291743) | 0.278204 / 0.434364 (-0.156160) | 0.326007 / 0.540337 (-0.214330) | 0.590881 / 1.386936 (-0.796055) |

</details>
</details>

![](https://cml.dev/watermark.png#c291e330a7d460ff09d867377de1d4c53fd5394c "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005266 / 0.011353 (-0.006087) | 0.003751 / 0.011008 (-0.007257) | 0.063835 / 0.038508 (0.025327) | 0.052688 / 0.023109 (0.029579) | 0.261957 / 0.275898 (-0.013941) | 0.284264 / 0.323480 (-0.039216) | 0.003958 / 0.007986 (-0.004027) | 0.002696 / 0.004328 (-0.001633) | 0.052791 / 0.004250 (0.048540) | 0.038294 / 0.037052 (0.001242) | 0.259488 / 0.258489 (0.000999) | 0.298368 / 0.293841 (0.004528) | 0.028309 / 0.128546 (-0.100237) | 0.010819 / 0.075646 (-0.064827) | 0.208221 / 0.419271 (-0.211050) | 0.036373 / 0.043533 (-0.007160) | 0.257000 / 0.255139 (0.001861) | 0.273108 / 0.283200 (-0.010092) | 0.019674 / 0.141683 (-0.122009) | 1.119196 / 1.452155 (-0.332958) | 1.161613 / 1.492716 (-0.331104) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093408 / 0.018006 (0.075401) | 0.302278 / 0.000490 (0.301788) | 0.000212 / 0.000200 (0.000012) | 0.000074 / 0.000054 (0.000020) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019417 / 0.037411 (-0.017995) | 0.060847 / 0.014526 (0.046321) | 0.075399 / 0.176557 (-0.101158) | 0.121233 / 0.737135 (-0.615902) | 0.076916 / 0.296338 (-0.219422) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.281265 / 0.215209 (0.066056) | 2.651726 / 2.077655 (0.574072) | 1.457726 / 1.504120 (-0.046394) | 1.339250 / 1.541195 (-0.201945) | 1.398529 / 1.468490 (-0.069961) | 0.566574 / 4.584777 (-4.018203) | 2.431576 / 3.745712 (-1.314136) | 2.845884 / 5.269862 (-2.423977) | 1.798051 / 4.565676 (-2.767626) | 0.063619 / 0.424275 (-0.360656) | 0.005286 / 0.007607 (-0.002321) | 0.332834 / 0.226044 (0.106789) | 3.293222 / 2.268929 (1.024293) | 1.837810 / 55.444624 (-53.606815) | 1.568511 / 6.876477 (-5.307966) | 1.627518 / 2.142072 (-0.514555) | 0.643520 / 4.805227 (-4.161708) | 0.118482 / 6.500664 (-6.382182) | 0.049563 / 0.075469 (-0.025906) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.947767 / 1.841788 (-0.894021) | 11.994999 / 8.074308 (3.920691) | 10.662651 / 10.191392 (0.471259) | 0.142070 / 0.680424 (-0.538354) | 0.014276 / 0.534201 (-0.519925) | 0.288455 / 0.579283 (-0.290828) | 0.266335 / 0.434364 (-0.168029) | 0.328455 / 0.540337 (-0.211883) | 0.440740 / 1.386936 (-0.946196) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005636 / 0.011353 (-0.005717) | 0.003664 / 0.011008 (-0.007344) | 0.050340 / 0.038508 (0.011832) | 0.062795 / 0.023109 (0.039685) | 0.280874 / 0.275898 (0.004976) | 0.314056 / 0.323480 (-0.009424) | 0.004089 / 0.007986 (-0.003897) | 0.002780 / 0.004328 (-0.001548) | 0.048468 / 0.004250 (0.044218) | 0.042924 / 0.037052 (0.005871) | 0.281381 / 0.258489 (0.022892) | 0.308232 / 0.293841 (0.014391) | 0.030294 / 0.128546 (-0.098252) | 0.011098 / 0.075646 (-0.064548) | 0.057535 / 0.419271 (-0.361736) | 0.034217 / 0.043533 (-0.009316) | 0.283022 / 0.255139 (0.027883) | 0.298425 / 0.283200 (0.015225) | 0.019285 / 0.141683 (-0.122398) | 1.117722 / 1.452155 (-0.334433) | 1.185878 / 1.492716 (-0.306839) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094915 / 0.018006 (0.076909) | 0.311782 / 0.000490 (0.311293) | 0.000217 / 0.000200 (0.000017) | 0.000054 / 0.000054 (-0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022652 / 0.037411 (-0.014759) | 0.069766 / 0.014526 (0.055240) | 0.084495 / 0.176557 (-0.092061) | 0.121295 / 0.737135 (-0.615841) | 0.082447 / 0.296338 (-0.213891) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.294286 / 0.215209 (0.079077) | 2.863694 / 2.077655 (0.786039) | 1.578338 / 1.504120 (0.074219) | 1.478737 / 1.541195 (-0.062458) | 1.528569 / 1.468490 (0.060079) | 0.576944 / 4.584777 (-4.007833) | 2.438730 / 3.745712 (-1.306982) | 2.956138 / 5.269862 (-2.313723) | 1.844484 / 4.565676 (-2.721192) | 0.065980 / 0.424275 (-0.358295) | 0.004998 / 0.007607 (-0.002609) | 0.352063 / 0.226044 (0.126019) | 3.456355 / 2.268929 (1.187426) | 1.971582 / 55.444624 (-53.473042) | 1.684536 / 6.876477 (-5.191940) | 1.726823 / 2.142072 (-0.415250) | 0.660235 / 4.805227 (-4.144992) | 0.119029 / 6.500664 (-6.381635) | 0.042497 / 0.075469 (-0.032972) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.971817 / 1.841788 (-0.869970) | 12.900324 / 8.074308 (4.826015) | 10.957495 / 10.191392 (0.766103) | 0.133705 / 0.680424 (-0.546718) | 0.015669 / 0.534201 (-0.518532) | 0.287340 / 0.579283 (-0.291943) | 0.280380 / 0.434364 (-0.153984) | 0.330369 / 0.540337 (-0.209969) | 0.581793 / 1.386936 (-0.805143) |

</details>
</details>

![](https://cml.dev/watermark.png#c2af5efae1985499d6a0a1b6ab4120337eebf776 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005038 / 0.011353 (-0.006315) | 0.003737 / 0.011008 (-0.007272) | 0.063118 / 0.038508 (0.024610) | 0.050120 / 0.023109 (0.027011) | 0.240722 / 0.275898 (-0.035176) | 0.263128 / 0.323480 (-0.060352) | 0.003839 / 0.007986 (-0.004147) | 0.002718 / 0.004328 (-0.001610) | 0.047869 / 0.004250 (0.043618) | 0.038092 / 0.037052 (0.001040) | 0.245759 / 0.258489 (-0.012730) | 0.277728 / 0.293841 (-0.016113) | 0.027466 / 0.128546 (-0.101081) | 0.011767 / 0.075646 (-0.063879) | 0.205505 / 0.419271 (-0.213766) | 0.035429 / 0.043533 (-0.008104) | 0.241665 / 0.255139 (-0.013474) | 0.260908 / 0.283200 (-0.022292) | 0.017133 / 0.141683 (-0.124550) | 1.107725 / 1.452155 (-0.344429) | 1.169707 / 1.492716 (-0.323009) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094112 / 0.018006 (0.076106) | 0.302596 / 0.000490 (0.302106) | 0.000237 / 0.000200 (0.000037) | 0.000041 / 0.000054 (-0.000013) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.017923 / 0.037411 (-0.019488) | 0.060356 / 0.014526 (0.045830) | 0.073708 / 0.176557 (-0.102849) | 0.119952 / 0.737135 (-0.617183) | 0.075350 / 0.296338 (-0.220989) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.289253 / 0.215209 (0.074044) | 2.800772 / 2.077655 (0.723117) | 1.538368 / 1.504120 (0.034248) | 1.401037 / 1.541195 (-0.140158) | 1.427170 / 1.468490 (-0.041320) | 0.560497 / 4.584777 (-4.024280) | 2.417844 / 3.745712 (-1.327868) | 2.798377 / 5.269862 (-2.471484) | 1.756517 / 4.565676 (-2.809160) | 0.063897 / 0.424275 (-0.360378) | 0.005323 / 0.007607 (-0.002284) | 0.339881 / 0.226044 (0.113836) | 3.354858 / 2.268929 (1.085929) | 1.877233 / 55.444624 (-53.567391) | 1.578713 / 6.876477 (-5.297764) | 1.631898 / 2.142072 (-0.510175) | 0.640303 / 4.805227 (-4.164924) | 0.116731 / 6.500664 (-6.383933) | 0.041978 / 0.075469 (-0.033491) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.963259 / 1.841788 (-0.878529) | 11.983646 / 8.074308 (3.909338) | 10.561596 / 10.191392 (0.370204) | 0.135863 / 0.680424 (-0.544561) | 0.015607 / 0.534201 (-0.518594) | 0.295164 / 0.579283 (-0.284119) | 0.283366 / 0.434364 (-0.150998) | 0.341848 / 0.540337 (-0.198489) | 0.448359 / 1.386936 (-0.938577) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005366 / 0.011353 (-0.005987) | 0.003621 / 0.011008 (-0.007387) | 0.048615 / 0.038508 (0.010107) | 0.053950 / 0.023109 (0.030841) | 0.273112 / 0.275898 (-0.002786) | 0.295655 / 0.323480 (-0.027825) | 0.004066 / 0.007986 (-0.003920) | 0.002700 / 0.004328 (-0.001628) | 0.047899 / 0.004250 (0.043648) | 0.041633 / 0.037052 (0.004581) | 0.277760 / 0.258489 (0.019271) | 0.302068 / 0.293841 (0.008227) | 0.028879 / 0.128546 (-0.099668) | 0.010756 / 0.075646 (-0.064891) | 0.057190 / 0.419271 (-0.362082) | 0.032555 / 0.043533 (-0.010978) | 0.272045 / 0.255139 (0.016906) | 0.289330 / 0.283200 (0.006130) | 0.018466 / 0.141683 (-0.123216) | 1.180435 / 1.452155 (-0.271720) | 1.192228 / 1.492716 (-0.300488) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094871 / 0.018006 (0.076864) | 0.302552 / 0.000490 (0.302062) | 0.000224 / 0.000200 (0.000024) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022008 / 0.037411 (-0.015403) | 0.068528 / 0.014526 (0.054002) | 0.081735 / 0.176557 (-0.094821) | 0.120990 / 0.737135 (-0.616145) | 0.083155 / 0.296338 (-0.213184) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.305030 / 0.215209 (0.089821) | 3.009812 / 2.077655 (0.932158) | 1.677773 / 1.504120 (0.173654) | 1.552280 / 1.541195 (0.011085) | 1.606248 / 1.468490 (0.137758) | 0.557093 / 4.584777 (-4.027684) | 2.418292 / 3.745712 (-1.327420) | 2.813049 / 5.269862 (-2.456813) | 1.764507 / 4.565676 (-2.801169) | 0.065089 / 0.424275 (-0.359186) | 0.004944 / 0.007607 (-0.002663) | 0.360672 / 0.226044 (0.134628) | 3.525850 / 2.268929 (1.256921) | 2.030091 / 55.444624 (-53.414533) | 1.754669 / 6.876477 (-5.121807) | 1.772673 / 2.142072 (-0.369399) | 0.642904 / 4.805227 (-4.162324) | 0.116018 / 6.500664 (-6.384646) | 0.041308 / 0.075469 (-0.034161) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.986386 / 1.841788 (-0.855401) | 12.291623 / 8.074308 (4.217315) | 10.655932 / 10.191392 (0.464540) | 0.141736 / 0.680424 (-0.538688) | 0.016669 / 0.534201 (-0.517532) | 0.286875 / 0.579283 (-0.292408) | 0.281898 / 0.434364 (-0.152466) | 0.325206 / 0.540337 (-0.215132) | 0.577607 / 1.386936 (-0.809329) |

</details>
</details>

![](https://cml.dev/watermark.png#1cf33502493fb9760ea8cc8e51622bf94d0c9e31 "CML watermark")


Alright tests are passing (except one on temp dir cleanup windows but I don't think it's related to this PR ?)

```
FAILED tests/test_load.py::test_loading_from_the_datasets_hub - NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpqy3f2ft_\\hf-internal-testing___dataset_with_script\\default\\0.0.0\\c240e2be3370bdbd\\dataset_with_script-train.arrow'
```

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005072 / 0.011353 (-0.006281) | 0.003449 / 0.011008 (-0.007559) | 0.062630 / 0.038508 (0.024122) | 0.054276 / 0.023109 (0.031167) | 0.253345 / 0.275898 (-0.022553) | 0.273460 / 0.323480 (-0.050020) | 0.003859 / 0.007986 (-0.004127) | 0.002646 / 0.004328 (-0.001683) | 0.048289 / 0.004250 (0.044038) | 0.037943 / 0.037052 (0.000891) | 0.256569 / 0.258489 (-0.001920) | 0.287809 / 0.293841 (-0.006032) | 0.027675 / 0.128546 (-0.100872) | 0.010554 / 0.075646 (-0.065092) | 0.205157 / 0.419271 (-0.214115) | 0.035464 / 0.043533 (-0.008069) | 0.254300 / 0.255139 (-0.000839) | 0.272907 / 0.283200 (-0.010292) | 0.018146 / 0.141683 (-0.123537) | 1.110528 / 1.452155 (-0.341626) | 1.170156 / 1.492716 (-0.322560) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093151 / 0.018006 (0.075144) | 0.302087 / 0.000490 (0.301598) | 0.000216 / 0.000200 (0.000016) | 0.000042 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018744 / 0.037411 (-0.018667) | 0.059843 / 0.014526 (0.045317) | 0.073165 / 0.176557 (-0.103391) | 0.120464 / 0.737135 (-0.616671) | 0.074992 / 0.296338 (-0.221347) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.285103 / 0.215209 (0.069894) | 2.820254 / 2.077655 (0.742600) | 1.505336 / 1.504120 (0.001216) | 1.368631 / 1.541195 (-0.172564) | 1.404140 / 1.468490 (-0.064350) | 0.563906 / 4.584777 (-4.020871) | 2.411871 / 3.745712 (-1.333841) | 2.788390 / 5.269862 (-2.481471) | 1.749788 / 4.565676 (-2.815888) | 0.062171 / 0.424275 (-0.362104) | 0.004918 / 0.007607 (-0.002689) | 0.339615 / 0.226044 (0.113571) | 3.337789 / 2.268929 (1.068861) | 1.808445 / 55.444624 (-53.636180) | 1.541015 / 6.876477 (-5.335462) | 1.572389 / 2.142072 (-0.569683) | 0.641739 / 4.805227 (-4.163488) | 0.115844 / 6.500664 (-6.384820) | 0.042504 / 0.075469 (-0.032965) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.942463 / 1.841788 (-0.899325) | 11.602364 / 8.074308 (3.528056) | 10.628921 / 10.191392 (0.437529) | 0.136154 / 0.680424 (-0.544270) | 0.013842 / 0.534201 (-0.520359) | 0.287085 / 0.579283 (-0.292198) | 0.269860 / 0.434364 (-0.164503) | 0.329525 / 0.540337 (-0.210812) | 0.441287 / 1.386936 (-0.945649) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005215 / 0.011353 (-0.006138) | 0.003549 / 0.011008 (-0.007460) | 0.049199 / 0.038508 (0.010691) | 0.051655 / 0.023109 (0.028545) | 0.272150 / 0.275898 (-0.003748) | 0.291978 / 0.323480 (-0.031502) | 0.003985 / 0.007986 (-0.004001) | 0.002668 / 0.004328 (-0.001661) | 0.048524 / 0.004250 (0.044274) | 0.039824 / 0.037052 (0.002772) | 0.275566 / 0.258489 (0.017077) | 0.298076 / 0.293841 (0.004235) | 0.029508 / 0.128546 (-0.099038) | 0.010673 / 0.075646 (-0.064973) | 0.057327 / 0.419271 (-0.361944) | 0.032590 / 0.043533 (-0.010943) | 0.273295 / 0.255139 (0.018156) | 0.289127 / 0.283200 (0.005928) | 0.017694 / 0.141683 (-0.123989) | 1.134502 / 1.452155 (-0.317653) | 1.185603 / 1.492716 (-0.307114) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.098403 / 0.018006 (0.080396) | 0.302735 / 0.000490 (0.302245) | 0.000228 / 0.000200 (0.000028) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025192 / 0.037411 (-0.012219) | 0.068149 / 0.014526 (0.053623) | 0.082220 / 0.176557 (-0.094336) | 0.119491 / 0.737135 (-0.617645) | 0.082484 / 0.296338 (-0.213855) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.295339 / 0.215209 (0.080130) | 2.868411 / 2.077655 (0.790757) | 1.590665 / 1.504120 (0.086545) | 1.465995 / 1.541195 (-0.075200) | 1.489205 / 1.468490 (0.020715) | 0.562503 / 4.584777 (-4.022274) | 2.480100 / 3.745712 (-1.265613) | 2.774216 / 5.269862 (-2.495646) | 1.733129 / 4.565676 (-2.832548) | 0.062698 / 0.424275 (-0.361577) | 0.004910 / 0.007607 (-0.002697) | 0.354766 / 0.226044 (0.128722) | 3.435541 / 2.268929 (1.166613) | 1.953357 / 55.444624 (-53.491267) | 1.673584 / 6.876477 (-5.202893) | 1.677749 / 2.142072 (-0.464323) | 0.632601 / 4.805227 (-4.172626) | 0.114875 / 6.500664 (-6.385789) | 0.040577 / 0.075469 (-0.034892) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.967003 / 1.841788 (-0.874785) | 11.964490 / 8.074308 (3.890181) | 10.493812 / 10.191392 (0.302420) | 0.132177 / 0.680424 (-0.548247) | 0.015149 / 0.534201 (-0.519052) | 0.289011 / 0.579283 (-0.290272) | 0.285479 / 0.434364 (-0.148885) | 0.327090 / 0.540337 (-0.213248) | 0.571747 / 1.386936 (-0.815189) |

</details>
</details>

![](https://cml.dev/watermark.png#4c9b4cb7ee4720415261216d72051e2a3320fe41 "CML watermark")