Release: 2.14.1
State: closed | Author: lhoestq (MEMBER)


Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6082). All of your documentation changes will be reflected on that endpoint.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007215 / 0.011353 (-0.004138) | 0.004101 / 0.011008 (-0.006907) | 0.085884 / 0.038508 (0.047376) | 0.085375 / 0.023109 (0.062266) | 0.351610 / 0.275898 (0.075712) | 0.399284 / 0.323480 (0.075804) | 0.005598 / 0.007986 (-0.002388) | 0.003405 / 0.004328 (-0.000923) | 0.064906 / 0.004250 (0.060656) | 0.059000 / 0.037052 (0.021948) | 0.354589 / 0.258489 (0.096100) | 0.406070 / 0.293841 (0.112229) | 0.031627 / 0.128546 (-0.096919) | 0.008597 / 0.075646 (-0.067049) | 0.291050 / 0.419271 (-0.128221) | 0.054120 / 0.043533 (0.010587) | 0.366242 / 0.255139 (0.111103) | 0.375975 / 0.283200 (0.092776) | 0.025608 / 0.141683 (-0.116074) | 1.473514 / 1.452155 (0.021359) | 1.543226 / 1.492716 (0.050510) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.198068 / 0.018006 (0.180062) | 0.450583 / 0.000490 (0.450093) | 0.005368 / 0.000200 (0.005168) | 0.000102 / 0.000054 (0.000047) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028323 / 0.037411 (-0.009089) | 0.089058 / 0.014526 (0.074533) | 0.097718 / 0.176557 (-0.078839) | 0.154546 / 0.737135 (-0.582590) | 0.098224 / 0.296338 (-0.198115) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.386292 / 0.215209 (0.171083) | 3.846222 / 2.077655 (1.768567) | 1.858695 / 1.504120 (0.354575) | 1.685885 / 1.541195 (0.144690) | 1.790727 / 1.468490 (0.322237) | 0.486771 / 4.584777 (-4.098006) | 3.658363 / 3.745712 (-0.087349) | 5.345236 / 5.269862 (0.075374) | 3.215942 / 4.565676 (-1.349734) | 0.057580 / 0.424275 (-0.366695) | 0.007382 / 0.007607 (-0.000225) | 0.464174 / 0.226044 (0.238129) | 4.640848 / 2.268929 (2.371920) | 2.383152 / 55.444624 (-53.061472) | 2.013288 / 6.876477 (-4.863188) | 2.244142 / 2.142072 (0.102069) | 0.585408 / 4.805227 (-4.219819) | 0.134698 / 6.500664 (-6.365966) | 0.060641 / 0.075469 (-0.014828) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.258414 / 1.841788 (-0.583374) | 19.825848 / 8.074308 (11.751540) | 14.644025 / 10.191392 (4.452633) | 0.169198 / 0.680424 (-0.511226) | 0.018180 / 0.534201 (-0.516021) | 0.395100 / 0.579283 (-0.184183) | 0.411543 / 0.434364 (-0.022821) | 0.463364 / 0.540337 (-0.076973) | 0.628613 / 1.386936 (-0.758323) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006860 / 0.011353 (-0.004493) | 0.003981 / 0.011008 (-0.007027) | 0.065589 / 0.038508 (0.027081) | 0.082460 / 0.023109 (0.059350) | 0.362980 / 0.275898 (0.087082) | 0.394837 / 0.323480 (0.071357) | 0.005298 / 0.007986 (-0.002688) | 0.003372 / 0.004328 (-0.000957) | 0.064918 / 0.004250 (0.060667) | 0.058033 / 0.037052 (0.020981) | 0.367259 / 0.258489 (0.108770) | 0.403122 / 0.293841 (0.109281) | 0.031566 / 0.128546 (-0.096980) | 0.008583 / 0.075646 (-0.067063) | 0.071287 / 0.419271 (-0.347984) | 0.049586 / 0.043533 (0.006053) | 0.359252 / 0.255139 (0.104113) | 0.378519 / 0.283200 (0.095319) | 0.023412 / 0.141683 (-0.118271) | 1.494522 / 1.452155 (0.042367) | 1.559176 / 1.492716 (0.066460) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.228396 / 0.018006 (0.210390) | 0.441865 / 0.000490 (0.441375) | 0.000395 / 0.000200 (0.000195) | 0.000054 / 0.000054 (-0.000000) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031169 / 0.037411 (-0.006242) | 0.093427 / 0.014526 (0.078901) | 0.100673 / 0.176557 (-0.075883) | 0.152817 / 0.737135 (-0.584319) | 0.102226 / 0.296338 (-0.194112) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.437032 / 0.215209 (0.221823) | 4.376078 / 2.077655 (2.298423) | 2.346928 / 1.504120 (0.842808) | 2.168573 / 1.541195 (0.627378) | 2.261024 / 1.468490 (0.792534) | 0.497080 / 4.584777 (-4.087697) | 3.594402 / 3.745712 (-0.151310) | 5.090361 / 5.269862 (-0.179501) | 3.034750 / 4.565676 (-1.530927) | 0.058538 / 0.424275 (-0.365737) | 0.007892 / 0.007607 (0.000285) | 0.517643 / 0.226044 (0.291598) | 5.173174 / 2.268929 (2.904246) | 2.825917 / 55.444624 (-52.618708) | 2.542593 / 6.876477 (-4.333884) | 2.716290 / 2.142072 (0.574218) | 0.598253 / 4.805227 (-4.206974) | 0.135610 / 6.500664 (-6.365054) | 0.062113 / 0.075469 (-0.013356) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.389554 / 1.841788 (-0.452233) | 20.412868 / 8.074308 (12.338560) | 14.539988 / 10.191392 (4.348596) | 0.162046 / 0.680424 (-0.518378) | 0.018508 / 0.534201 (-0.515693) | 0.398840 / 0.579283 (-0.180443) | 0.400902 / 0.434364 (-0.033462) | 0.463647 / 0.540337 (-0.076691) | 0.612921 / 1.386936 (-0.774015) |

</details>
</details>

![](https://cml.dev/watermark.png#45bef1810d9341ba4cb27547d748fddb97843792 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005943 / 0.011353 (-0.005410) | 0.003582 / 0.011008 (-0.007426) | 0.080030 / 0.038508 (0.041522) | 0.057458 / 0.023109 (0.034349) | 0.390783 / 0.275898 (0.114885) | 0.430926 / 0.323480 (0.107446) | 0.003207 / 0.007986 (-0.004778) | 0.003592 / 0.004328 (-0.000737) | 0.062468 / 0.004250 (0.058217) | 0.046739 / 0.037052 (0.009687) | 0.394343 / 0.258489 (0.135854) | 0.435912 / 0.293841 (0.142071) | 0.026812 / 0.128546 (-0.101734) | 0.007954 / 0.075646 (-0.067692) | 0.261415 / 0.419271 (-0.157857) | 0.044665 / 0.043533 (0.001132) | 0.403454 / 0.255139 (0.148315) | 0.418946 / 0.283200 (0.135747) | 0.022247 / 0.141683 (-0.119436) | 1.456387 / 1.452155 (0.004232) | 1.508234 / 1.492716 (0.015518) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.182487 / 0.018006 (0.164480) | 0.416343 / 0.000490 (0.415854) | 0.001404 / 0.000200 (0.001204) | 0.000062 / 0.000054 (0.000007) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023643 / 0.037411 (-0.013768) | 0.071798 / 0.014526 (0.057272) | 0.083623 / 0.176557 (-0.092933) | 0.146023 / 0.737135 (-0.591112) | 0.083094 / 0.296338 (-0.213245) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.417484 / 0.215209 (0.202275) | 4.157393 / 2.077655 (2.079738) | 1.950438 / 1.504120 (0.446318) | 1.766639 / 1.541195 (0.225444) | 1.807382 / 1.468490 (0.338892) | 0.496061 / 4.584777 (-4.088716) | 2.975001 / 3.745712 (-0.770711) | 3.340608 / 5.269862 (-1.929254) | 2.236293 / 4.565676 (-2.329384) | 0.056946 / 0.424275 (-0.367329) | 0.006506 / 0.007607 (-0.001101) | 0.480377 / 0.226044 (0.254332) | 4.788525 / 2.268929 (2.519597) | 2.430139 / 55.444624 (-53.014485) | 2.154145 / 6.876477 (-4.722332) | 2.321623 / 2.142072 (0.179551) | 0.584040 / 4.805227 (-4.221188) | 0.124508 / 6.500664 (-6.376156) | 0.060828 / 0.075469 (-0.014641) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.201641 / 1.841788 (-0.640146) | 18.066232 / 8.074308 (9.991924) | 14.022304 / 10.191392 (3.830912) | 0.146573 / 0.680424 (-0.533850) | 0.016892 / 0.534201 (-0.517308) | 0.333259 / 0.579283 (-0.246024) | 0.357795 / 0.434364 (-0.076568) | 0.391265 / 0.540337 (-0.149072) | 0.551378 / 1.386936 (-0.835558) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005706 / 0.011353 (-0.005647) | 0.003448 / 0.011008 (-0.007560) | 0.063146 / 0.038508 (0.024638) | 0.056292 / 0.023109 (0.033183) | 0.355533 / 0.275898 (0.079635) | 0.394996 / 0.323480 (0.071517) | 0.004270 / 0.007986 (-0.003716) | 0.002790 / 0.004328 (-0.001538) | 0.063033 / 0.004250 (0.058783) | 0.044684 / 0.037052 (0.007631) | 0.370621 / 0.258489 (0.112132) | 0.401074 / 0.293841 (0.107233) | 0.026737 / 0.128546 (-0.101809) | 0.007872 / 0.075646 (-0.067774) | 0.068815 / 0.419271 (-0.350457) | 0.040976 / 0.043533 (-0.002557) | 0.370733 / 0.255139 (0.115594) | 0.387418 / 0.283200 (0.104218) | 0.018854 / 0.141683 (-0.122829) | 1.479834 / 1.452155 (0.027680) | 1.536388 / 1.492716 (0.043672) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.222125 / 0.018006 (0.204119) | 0.408007 / 0.000490 (0.407517) | 0.000367 / 0.000200 (0.000167) | 0.000055 / 0.000054 (0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025100 / 0.037411 (-0.012311) | 0.076617 / 0.014526 (0.062091) | 0.088311 / 0.176557 (-0.088246) | 0.143785 / 0.737135 (-0.593350) | 0.088349 / 0.296338 (-0.207989) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.419246 / 0.215209 (0.204037) | 4.172413 / 2.077655 (2.094759) | 2.199355 / 1.504120 (0.695235) | 2.025158 / 1.541195 (0.483963) | 2.074491 / 1.468490 (0.606001) | 0.495893 / 4.584777 (-4.088884) | 2.998858 / 3.745712 (-0.746854) | 2.770531 / 5.269862 (-2.499331) | 1.817497 / 4.565676 (-2.748179) | 0.057317 / 0.424275 (-0.366958) | 0.006723 / 0.007607 (-0.000884) | 0.491062 / 0.226044 (0.265017) | 4.906155 / 2.268929 (2.637226) | 2.654916 / 55.444624 (-52.789708) | 2.299873 / 6.876477 (-4.576604) | 2.451438 / 2.142072 (0.309366) | 0.585048 / 4.805227 (-4.220179) | 0.124778 / 6.500664 (-6.375886) | 0.062067 / 0.075469 (-0.013402) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.298239 / 1.841788 (-0.543549) | 18.090238 / 8.074308 (10.015930) | 13.822568 / 10.191392 (3.631176) | 0.130560 / 0.680424 (-0.549864) | 0.016662 / 0.534201 (-0.517539) | 0.333337 / 0.579283 (-0.245946) | 0.348493 / 0.434364 (-0.085871) | 0.386049 / 0.540337 (-0.154289) | 0.511156 / 1.386936 (-0.875780) |

</details>
</details>

![](https://cml.dev/watermark.png#029956a347b0306cd27f693e12cf9a82acf4ef80 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006014 / 0.011353 (-0.005339) | 0.003623 / 0.011008 (-0.007385) | 0.080500 / 0.038508 (0.041992) | 0.057713 / 0.023109 (0.034603) | 0.325976 / 0.275898 (0.050078) | 0.359986 / 0.323480 (0.036506) | 0.004709 / 0.007986 (-0.003277) | 0.002933 / 0.004328 (-0.001395) | 0.063457 / 0.004250 (0.059207) | 0.047514 / 0.037052 (0.010462) | 0.331629 / 0.258489 (0.073140) | 0.382048 / 0.293841 (0.088207) | 0.026949 / 0.128546 (-0.101597) | 0.008043 / 0.075646 (-0.067604) | 0.262152 / 0.419271 (-0.157119) | 0.045271 / 0.043533 (0.001738) | 0.333355 / 0.255139 (0.078216) | 0.347996 / 0.283200 (0.064796) | 0.020814 / 0.141683 (-0.120868) | 1.460723 / 1.452155 (0.008568) | 1.488845 / 1.492716 (-0.003872) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.193735 / 0.018006 (0.175728) | 0.431433 / 0.000490 (0.430943) | 0.002494 / 0.000200 (0.002294) | 0.000066 / 0.000054 (0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023762 / 0.037411 (-0.013650) | 0.072680 / 0.014526 (0.058154) | 0.081687 / 0.176557 (-0.094869) | 0.143224 / 0.737135 (-0.593911) | 0.083083 / 0.296338 (-0.213255) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.397393 / 0.215209 (0.182184) | 3.954643 / 2.077655 (1.876989) | 1.950038 / 1.504120 (0.445919) | 1.760551 / 1.541195 (0.219357) | 1.871165 / 1.468490 (0.402675) | 0.508645 / 4.584777 (-4.076132) | 3.114379 / 3.745712 (-0.631333) | 3.474554 / 5.269862 (-1.795307) | 2.090126 / 4.565676 (-2.475551) | 0.058008 / 0.424275 (-0.366267) | 0.006465 / 0.007607 (-0.001142) | 0.475009 / 0.226044 (0.248965) | 4.767981 / 2.268929 (2.499052) | 2.372050 / 55.444624 (-53.072574) | 2.038094 / 6.876477 (-4.838383) | 2.072819 / 2.142072 (-0.069253) | 0.591913 / 4.805227 (-4.213314) | 0.125002 / 6.500664 (-6.375662) | 0.060055 / 0.075469 (-0.015414) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.234171 / 1.841788 (-0.607617) | 18.121476 / 8.074308 (10.047168) | 13.727313 / 10.191392 (3.535921) | 0.136021 / 0.680424 (-0.544402) | 0.016505 / 0.534201 (-0.517696) | 0.331400 / 0.579283 (-0.247883) | 0.346019 / 0.434364 (-0.088345) | 0.378985 / 0.540337 (-0.161353) | 0.522606 / 1.386936 (-0.864330) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006035 / 0.011353 (-0.005318) | 0.003584 / 0.011008 (-0.007425) | 0.061953 / 0.038508 (0.023445) | 0.059416 / 0.023109 (0.036307) | 0.359380 / 0.275898 (0.083482) | 0.396842 / 0.323480 (0.073363) | 0.004716 / 0.007986 (-0.003269) | 0.002825 / 0.004328 (-0.001504) | 0.061697 / 0.004250 (0.057447) | 0.049009 / 0.037052 (0.011956) | 0.363099 / 0.258489 (0.104610) | 0.403672 / 0.293841 (0.109831) | 0.027722 / 0.128546 (-0.100824) | 0.007966 / 0.075646 (-0.067680) | 0.067455 / 0.419271 (-0.351816) | 0.042530 / 0.043533 (-0.001003) | 0.361257 / 0.255139 (0.106118) | 0.388957 / 0.283200 (0.105758) | 0.021845 / 0.141683 (-0.119838) | 1.431989 / 1.452155 (-0.020166) | 1.503131 / 1.492716 (0.010415) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.241493 / 0.018006 (0.223487) | 0.429319 / 0.000490 (0.428829) | 0.002604 / 0.000200 (0.002404) | 0.000074 / 0.000054 (0.000019) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026227 / 0.037411 (-0.011184) | 0.077177 / 0.014526 (0.062651) | 0.085840 / 0.176557 (-0.090717) | 0.142280 / 0.737135 (-0.594855) | 0.088465 / 0.296338 (-0.207873) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.434912 / 0.215209 (0.219703) | 4.339664 / 2.077655 (2.262009) | 2.242495 / 1.504120 (0.738375) | 2.091353 / 1.541195 (0.550159) | 2.161425 / 1.468490 (0.692935) | 0.501647 / 4.584777 (-4.083130) | 3.075326 / 3.745712 (-0.670386) | 4.091557 / 5.269862 (-1.178304) | 2.776425 / 4.565676 (-1.789251) | 0.057338 / 0.424275 (-0.366937) | 0.006767 / 0.007607 (-0.000840) | 0.506882 / 0.226044 (0.280837) | 5.059074 / 2.268929 (2.790146) | 2.706665 / 55.444624 (-52.737959) | 2.370253 / 6.876477 (-4.506224) | 2.505421 / 2.142072 (0.363348) | 0.590289 / 4.805227 (-4.214938) | 0.125990 / 6.500664 (-6.374674) | 0.062778 / 0.075469 (-0.012691) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.361287 / 1.841788 (-0.480501) | 18.500726 / 8.074308 (10.426418) | 13.844459 / 10.191392 (3.653067) | 0.144416 / 0.680424 (-0.536008) | 0.016987 / 0.534201 (-0.517214) | 0.336237 / 0.579283 (-0.243046) | 0.357116 / 0.434364 (-0.077248) | 0.402062 / 0.540337 (-0.138275) | 0.543066 / 1.386936 (-0.843870) |

</details>
</details>

![](https://cml.dev/watermark.png#029956a347b0306cd27f693e12cf9a82acf4ef80 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007559 / 0.011353 (-0.003794) | 0.004379 / 0.011008 (-0.006629) | 0.089702 / 0.038508 (0.051194) | 0.065104 / 0.023109 (0.041995) | 0.362016 / 0.275898 (0.086118) | 0.376768 / 0.323480 (0.053288) | 0.006538 / 0.007986 (-0.001447) | 0.004167 / 0.004328 (-0.000161) | 0.074138 / 0.004250 (0.069888) | 0.052753 / 0.037052 (0.015701) | 0.366367 / 0.258489 (0.107878) | 0.389121 / 0.293841 (0.095280) | 0.042820 / 0.128546 (-0.085727) | 0.012560 / 0.075646 (-0.063086) | 0.359235 / 0.419271 (-0.060037) | 0.074250 / 0.043533 (0.030718) | 0.384051 / 0.255139 (0.128912) | 0.385450 / 0.283200 (0.102250) | 0.046270 / 0.141683 (-0.095413) | 1.593275 / 1.452155 (0.141120) | 1.704207 / 1.492716 (0.211490) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.249390 / 0.018006 (0.231384) | 0.614347 / 0.000490 (0.613857) | 0.012641 / 0.000200 (0.012441) | 0.000126 / 0.000054 (0.000072) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029099 / 0.037411 (-0.008312) | 0.090966 / 0.014526 (0.076440) | 0.102273 / 0.176557 (-0.074284) | 0.167564 / 0.737135 (-0.569571) | 0.106118 / 0.296338 (-0.190220) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.536122 / 0.215209 (0.320913) | 5.448464 / 2.077655 (3.370809) | 2.461977 / 1.504120 (0.957857) | 2.081506 / 1.541195 (0.540311) | 2.091509 / 1.468490 (0.623019) | 0.810307 / 4.584777 (-3.774470) | 5.161304 / 3.745712 (1.415592) | 4.525070 / 5.269862 (-0.744792) | 2.886313 / 4.565676 (-1.679363) | 0.093992 / 0.424275 (-0.330283) | 0.008516 / 0.007607 (0.000909) | 0.691978 / 0.226044 (0.465934) | 6.834665 / 2.268929 (4.565737) | 3.284355 / 55.444624 (-52.160270) | 2.496803 / 6.876477 (-4.379674) | 2.814387 / 2.142072 (0.672315) | 0.985300 / 4.805227 (-3.819928) | 0.210343 / 6.500664 (-6.290321) | 0.075459 / 0.075469 (-0.000010) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.436073 / 1.841788 (-0.405714) | 22.722401 / 8.074308 (14.648093) | 19.988521 / 10.191392 (9.797129) | 0.229757 / 0.680424 (-0.450667) | 0.029672 / 0.534201 (-0.504529) | 0.479914 / 0.579283 (-0.099369) | 0.605106 / 0.434364 (0.170743) | 0.511668 / 0.540337 (-0.028670) | 0.800281 / 1.386936 (-0.586655) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008665 / 0.011353 (-0.002688) | 0.006009 / 0.011008 (-0.004999) | 0.073377 / 0.038508 (0.034869) | 0.077188 / 0.023109 (0.054079) | 0.451422 / 0.275898 (0.175524) | 0.484640 / 0.323480 (0.161160) | 0.006266 / 0.007986 (-0.001719) | 0.004129 / 0.004328 (-0.000200) | 0.063102 / 0.004250 (0.058851) | 0.064653 / 0.037052 (0.027601) | 0.439521 / 0.258489 (0.181032) | 0.458964 / 0.293841 (0.165123) | 0.046018 / 0.128546 (-0.082528) | 0.014109 / 0.075646 (-0.061537) | 0.095727 / 0.419271 (-0.323544) | 0.070133 / 0.043533 (0.026600) | 0.440143 / 0.255139 (0.185004) | 0.502468 / 0.283200 (0.219269) | 0.034582 / 0.141683 (-0.107101) | 1.656282 / 1.452155 (0.204127) | 1.784641 / 1.492716 (0.291925) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.303111 / 0.018006 (0.285105) | 0.599194 / 0.000490 (0.598705) | 0.000411 / 0.000200 (0.000211) | 0.000073 / 0.000054 (0.000018) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033061 / 0.037411 (-0.004350) | 0.096073 / 0.014526 (0.081548) | 0.095347 / 0.176557 (-0.081209) | 0.161004 / 0.737135 (-0.576131) | 0.111544 / 0.296338 (-0.184794) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.615695 / 0.215209 (0.400486) | 5.794243 / 2.077655 (3.716588) | 2.594720 / 1.504120 (1.090600) | 2.566255 / 1.541195 (1.025060) | 2.573653 / 1.468490 (1.105163) | 0.873653 / 4.584777 (-3.711124) | 5.353323 / 3.745712 (1.607611) | 4.604974 / 5.269862 (-0.664887) | 2.901282 / 4.565676 (-1.664394) | 0.099614 / 0.424275 (-0.324661) | 0.010368 / 0.007607 (0.002761) | 0.775490 / 0.226044 (0.549446) | 7.245449 / 2.268929 (4.976520) | 3.740165 / 55.444624 (-51.704459) | 2.986132 / 6.876477 (-3.890345) | 3.092510 / 2.142072 (0.950438) | 1.022461 / 4.805227 (-3.782766) | 0.212137 / 6.500664 (-6.288527) | 0.084534 / 0.075469 (0.009065) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.687983 / 1.841788 (-0.153805) | 23.491808 / 8.074308 (15.417500) | 20.722165 / 10.191392 (10.530773) | 0.231011 / 0.680424 (-0.449413) | 0.028309 / 0.534201 (-0.505892) | 0.436911 / 0.579283 (-0.142372) | 0.583126 / 0.434364 (0.148762) | 0.559712 / 0.540337 (0.019374) | 0.820645 / 1.386936 (-0.566291) |

</details>
</details>

![](https://cml.dev/watermark.png#029956a347b0306cd27f693e12cf9a82acf4ef80 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006538 / 0.011353 (-0.004815) | 0.003952 / 0.011008 (-0.007056) | 0.084183 / 0.038508 (0.045675) | 0.070616 / 0.023109 (0.047507) | 0.320491 / 0.275898 (0.044593) | 0.352021 / 0.323480 (0.028541) | 0.005330 / 0.007986 (-0.002656) | 0.003400 / 0.004328 (-0.000928) | 0.066392 / 0.004250 (0.062141) | 0.052529 / 0.037052 (0.015477) | 0.329581 / 0.258489 (0.071092) | 0.374437 / 0.293841 (0.080596) | 0.031379 / 0.128546 (-0.097167) | 0.008576 / 0.075646 (-0.067070) | 0.288621 / 0.419271 (-0.130650) | 0.052748 / 0.043533 (0.009215) | 0.319911 / 0.255139 (0.064772) | 0.358169 / 0.283200 (0.074970) | 0.023128 / 0.141683 (-0.118555) | 1.479578 / 1.452155 (0.027424) | 1.566351 / 1.492716 (0.073635) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.217616 / 0.018006 (0.199610) | 0.471546 / 0.000490 (0.471056) | 0.003880 / 0.000200 (0.003680) | 0.000085 / 0.000054 (0.000031) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027716 / 0.037411 (-0.009696) | 0.081718 / 0.014526 (0.067192) | 0.095457 / 0.176557 (-0.081100) | 0.150746 / 0.737135 (-0.586389) | 0.096061 / 0.296338 (-0.200277) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.406811 / 0.215209 (0.191602) | 4.062757 / 2.077655 (1.985103) | 2.060658 / 1.504120 (0.556538) | 1.870944 / 1.541195 (0.329749) | 1.908984 / 1.468490 (0.440493) | 0.489053 / 4.584777 (-4.095724) | 3.571038 / 3.745712 (-0.174674) | 3.255351 / 5.269862 (-2.014511) | 2.007078 / 4.565676 (-2.558599) | 0.057078 / 0.424275 (-0.367197) | 0.007240 / 0.007607 (-0.000367) | 0.485641 / 0.226044 (0.259596) | 4.841657 / 2.268929 (2.572729) | 2.569676 / 55.444624 (-52.874949) | 2.151119 / 6.876477 (-4.725357) | 2.330337 / 2.142072 (0.188265) | 0.581721 / 4.805227 (-4.223506) | 0.132591 / 6.500664 (-6.368073) | 0.060491 / 0.075469 (-0.014978) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.237699 / 1.841788 (-0.604089) | 19.460306 / 8.074308 (11.385998) | 14.123006 / 10.191392 (3.931614) | 0.155669 / 0.680424 (-0.524754) | 0.018385 / 0.534201 (-0.515816) | 0.393330 / 0.579283 (-0.185953) | 0.408890 / 0.434364 (-0.025474) | 0.457348 / 0.540337 (-0.082989) | 0.640293 / 1.386936 (-0.746643) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006582 / 0.011353 (-0.004771) | 0.003950 / 0.011008 (-0.007059) | 0.064636 / 0.038508 (0.026128) | 0.077651 / 0.023109 (0.054541) | 0.365505 / 0.275898 (0.089607) | 0.393370 / 0.323480 (0.069890) | 0.005466 / 0.007986 (-0.002520) | 0.003314 / 0.004328 (-0.001014) | 0.064960 / 0.004250 (0.060710) | 0.057355 / 0.037052 (0.020302) | 0.377773 / 0.258489 (0.119284) | 0.408394 / 0.293841 (0.114553) | 0.031698 / 0.128546 (-0.096848) | 0.008575 / 0.075646 (-0.067071) | 0.070390 / 0.419271 (-0.348881) | 0.050035 / 0.043533 (0.006502) | 0.360461 / 0.255139 (0.105323) | 0.384862 / 0.283200 (0.101662) | 0.025380 / 0.141683 (-0.116303) | 1.484429 / 1.452155 (0.032275) | 1.542944 / 1.492716 (0.050227) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.190193 / 0.018006 (0.172187) | 0.468996 / 0.000490 (0.468506) | 0.003012 / 0.000200 (0.002812) | 0.000091 / 0.000054 (0.000036) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031488 / 0.037411 (-0.005923) | 0.088673 / 0.014526 (0.074147) | 0.101886 / 0.176557 (-0.074670) | 0.156774 / 0.737135 (-0.580361) | 0.102818 / 0.296338 (-0.193520) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.428019 / 0.215209 (0.212810) | 4.271369 / 2.077655 (2.193714) | 2.271530 / 1.504120 (0.767410) | 2.085172 / 1.541195 (0.543977) | 2.143439 / 1.468490 (0.674949) | 0.493468 / 4.584777 (-4.091309) | 3.569030 / 3.745712 (-0.176683) | 4.777962 / 5.269862 (-0.491900) | 2.872115 / 4.565676 (-1.693562) | 0.058200 / 0.424275 (-0.366075) | 0.007657 / 0.007607 (0.000050) | 0.502874 / 0.226044 (0.276830) | 5.026721 / 2.268929 (2.757792) | 2.734301 / 55.444624 (-52.710324) | 2.396072 / 6.876477 (-4.480405) | 2.574322 / 2.142072 (0.432249) | 0.593855 / 4.805227 (-4.211373) | 0.135134 / 6.500664 (-6.365530) | 0.061491 / 0.075469 (-0.013978) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.320522 / 1.841788 (-0.521265) | 19.933221 / 8.074308 (11.858912) | 14.055921 / 10.191392 (3.864529) | 0.149620 / 0.680424 (-0.530804) | 0.018590 / 0.534201 (-0.515611) | 0.399550 / 0.579283 (-0.179733) | 0.410463 / 0.434364 (-0.023901) | 0.469872 / 0.540337 (-0.070465) | 0.616481 / 1.386936 (-0.770455) |

</details>
</details>

![](https://cml.dev/watermark.png#029956a347b0306cd27f693e12cf9a82acf4ef80 "CML watermark")