Check if column names match in Parquet loader only when config `features` are specified
State: closed | Author: mariosasko (COLLABORATOR)

Fix #6039 

Comments: _The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006557 / 0.011353 (-0.004796) | 0.004096 / 0.011008 (-0.006913) | 0.083577 / 0.038508 (0.045069) | 0.072092 / 0.023109 (0.048983) | 0.319192 / 0.275898 (0.043294) | 0.351845 / 0.323480 (0.028365) | 0.005475 / 0.007986 (-0.002511) | 0.003419 / 0.004328 (-0.000910) | 0.064562 / 0.004250 (0.060311) | 0.057930 / 0.037052 (0.020878) | 0.326085 / 0.258489 (0.067596) | 0.368316 / 0.293841 (0.074475) | 0.030502 / 0.128546 (-0.098044) | 0.008504 / 0.075646 (-0.067142) | 0.287217 / 0.419271 (-0.132054) | 0.052337 / 0.043533 (0.008804) | 0.319011 / 0.255139 (0.063872) | 0.352711 / 0.283200 (0.069511) | 0.023278 / 0.141683 (-0.118405) | 1.482578 / 1.452155 (0.030423) | 1.553391 / 1.492716 (0.060675) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.199628 / 0.018006 (0.181622) | 0.464571 / 0.000490 (0.464081) | 0.003512 / 0.000200 (0.003312) | 0.000072 / 0.000054 (0.000018) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029109 / 0.037411 (-0.008302) | 0.082203 / 0.014526 (0.067677) | 0.096223 / 0.176557 (-0.080333) | 0.155598 / 0.737135 (-0.581537) | 0.097738 / 0.296338 (-0.198600) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.386135 / 0.215209 (0.170926) | 3.837157 / 2.077655 (1.759502) | 1.836869 / 1.504120 (0.332750) | 1.680592 / 1.541195 (0.139398) | 1.769456 / 1.468490 (0.300966) | 0.493150 / 4.584777 (-4.091627) | 3.589797 / 3.745712 (-0.155915) | 3.330000 / 5.269862 (-1.939861) | 2.059856 / 4.565676 (-2.505821) | 0.057951 / 0.424275 (-0.366324) | 0.007340 / 0.007607 (-0.000267) | 0.463203 / 0.226044 (0.237159) | 4.631514 / 2.268929 (2.362585) | 2.329887 / 55.444624 (-53.114738) | 2.008815 / 6.876477 (-4.867662) | 2.199067 / 2.142072 (0.056995) | 0.591417 / 4.805227 (-4.213810) | 0.137154 / 6.500664 (-6.363510) | 0.061326 / 0.075469 (-0.014143) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.269676 / 1.841788 (-0.572111) | 19.375167 / 8.074308 (11.300858) | 13.945419 / 10.191392 (3.754027) | 0.146482 / 0.680424 (-0.533942) | 0.018257 / 0.534201 (-0.515944) | 0.391684 / 0.579283 (-0.187599) | 0.411454 / 0.434364 (-0.022910) | 0.466260 / 0.540337 (-0.074077) | 0.655571 / 1.386936 (-0.731365) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006619 / 0.011353 (-0.004734) | 0.004102 / 0.011008 (-0.006907) | 0.064848 / 0.038508 (0.026340) | 0.074822 / 0.023109 (0.051713) | 0.366535 / 0.275898 (0.090637) | 0.395873 / 0.323480 (0.072394) | 0.005315 / 0.007986 (-0.002670) | 0.003270 / 0.004328 (-0.001059) | 0.064829 / 0.004250 (0.060578) | 0.056094 / 0.037052 (0.019042) | 0.370355 / 0.258489 (0.111866) | 0.406837 / 0.293841 (0.112996) | 0.031634 / 0.128546 (-0.096912) | 0.008569 / 0.075646 (-0.067077) | 0.071126 / 0.419271 (-0.348145) | 0.048629 / 0.043533 (0.005096) | 0.365175 / 0.255139 (0.110036) | 0.385234 / 0.283200 (0.102034) | 0.023295 / 0.141683 (-0.118388) | 1.466907 / 1.452155 (0.014752) | 1.523118 / 1.492716 (0.030401) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.227872 / 0.018006 (0.209866) | 0.451573 / 0.000490 (0.451083) | 0.000379 / 0.000200 (0.000179) | 0.000055 / 0.000054 (0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029496 / 0.037411 (-0.007915) | 0.086614 / 0.014526 (0.072088) | 0.098165 / 0.176557 (-0.078392) | 0.152218 / 0.737135 (-0.584917) | 0.101215 / 0.296338 (-0.195123) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.407519 / 0.215209 (0.192310) | 4.074704 / 2.077655 (1.997049) | 2.113185 / 1.504120 (0.609065) | 1.947461 / 1.541195 (0.406266) | 1.998521 / 1.468490 (0.530031) | 0.487463 / 4.584777 (-4.097313) | 3.465423 / 3.745712 (-0.280289) | 3.376498 / 5.269862 (-1.893363) | 2.001533 / 4.565676 (-2.564144) | 0.057052 / 0.424275 (-0.367223) | 0.007325 / 0.007607 (-0.000283) | 0.485648 / 0.226044 (0.259604) | 4.860191 / 2.268929 (2.591262) | 2.550340 / 55.444624 (-52.894284) | 2.231136 / 6.876477 (-4.645341) | 2.262539 / 2.142072 (0.120467) | 0.591422 / 4.805227 (-4.213805) | 0.132875 / 6.500664 (-6.367789) | 0.062154 / 0.075469 (-0.013315) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.321834 / 1.841788 (-0.519954) | 19.734750 / 8.074308 (11.660442) | 14.681049 / 10.191392 (4.489657) | 0.148894 / 0.680424 (-0.531530) | 0.018414 / 0.534201 (-0.515787) | 0.393377 / 0.579283 (-0.185906) | 0.402795 / 0.434364 (-0.031569) | 0.478624 / 0.540337 (-0.061714) | 0.656767 / 1.386936 (-0.730169) |

</details>
</details>

![](https://cml.dev/watermark.png#a5a84a1fa226a4cafb3bb4387dc4b212a46caf31 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007012 / 0.011353 (-0.004341) | 0.004120 / 0.011008 (-0.006888) | 0.083720 / 0.038508 (0.045212) | 0.083105 / 0.023109 (0.059996) | 0.323803 / 0.275898 (0.047905) | 0.340345 / 0.323480 (0.016865) | 0.005872 / 0.007986 (-0.002113) | 0.003528 / 0.004328 (-0.000801) | 0.065185 / 0.004250 (0.060935) | 0.063092 / 0.037052 (0.026040) | 0.314900 / 0.258489 (0.056411) | 0.349251 / 0.293841 (0.055410) | 0.031612 / 0.128546 (-0.096934) | 0.008541 / 0.075646 (-0.067105) | 0.289865 / 0.419271 (-0.129407) | 0.055264 / 0.043533 (0.011731) | 0.309152 / 0.255139 (0.054013) | 0.332625 / 0.283200 (0.049425) | 0.024306 / 0.141683 (-0.117377) | 1.489191 / 1.452155 (0.037037) | 1.562447 / 1.492716 (0.069731) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.236681 / 0.018006 (0.218675) | 0.567767 / 0.000490 (0.567277) | 0.003022 / 0.000200 (0.002822) | 0.000218 / 0.000054 (0.000164) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028698 / 0.037411 (-0.008714) | 0.081681 / 0.014526 (0.067155) | 0.099109 / 0.176557 (-0.077447) | 0.154381 / 0.737135 (-0.582754) | 0.098691 / 0.296338 (-0.197648) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.397985 / 0.215209 (0.182776) | 3.962499 / 2.077655 (1.884844) | 1.936158 / 1.504120 (0.432038) | 1.762339 / 1.541195 (0.221144) | 1.837451 / 1.468490 (0.368961) | 0.485655 / 4.584777 (-4.099122) | 3.538341 / 3.745712 (-0.207371) | 5.110095 / 5.269862 (-0.159767) | 3.066152 / 4.565676 (-1.499524) | 0.057505 / 0.424275 (-0.366770) | 0.007334 / 0.007607 (-0.000273) | 0.475622 / 0.226044 (0.249578) | 4.754091 / 2.268929 (2.485162) | 2.431379 / 55.444624 (-53.013246) | 2.106178 / 6.876477 (-4.770298) | 2.364305 / 2.142072 (0.222232) | 0.614038 / 4.805227 (-4.191190) | 0.148530 / 6.500664 (-6.352134) | 0.061033 / 0.075469 (-0.014436) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.242345 / 1.841788 (-0.599443) | 19.017266 / 8.074308 (10.942958) | 13.477782 / 10.191392 (3.286390) | 0.158513 / 0.680424 (-0.521911) | 0.018757 / 0.534201 (-0.515444) | 0.393773 / 0.579283 (-0.185510) | 0.416933 / 0.434364 (-0.017431) | 0.460012 / 0.540337 (-0.080326) | 0.637010 / 1.386936 (-0.749926) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006689 / 0.011353 (-0.004664) | 0.004168 / 0.011008 (-0.006840) | 0.065009 / 0.038508 (0.026501) | 0.073766 / 0.023109 (0.050657) | 0.369585 / 0.275898 (0.093687) | 0.407945 / 0.323480 (0.084465) | 0.005583 / 0.007986 (-0.002403) | 0.003494 / 0.004328 (-0.000835) | 0.065032 / 0.004250 (0.060782) | 0.057166 / 0.037052 (0.020114) | 0.370656 / 0.258489 (0.112166) | 0.428381 / 0.293841 (0.134540) | 0.031653 / 0.128546 (-0.096893) | 0.008731 / 0.075646 (-0.066915) | 0.071624 / 0.419271 (-0.347648) | 0.049364 / 0.043533 (0.005832) | 0.361824 / 0.255139 (0.106685) | 0.387615 / 0.283200 (0.104415) | 0.023228 / 0.141683 (-0.118455) | 1.476204 / 1.452155 (0.024049) | 1.553522 / 1.492716 (0.060806) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.266955 / 0.018006 (0.248948) | 0.556566 / 0.000490 (0.556076) | 0.000399 / 0.000200 (0.000199) | 0.000056 / 0.000054 (0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033104 / 0.037411 (-0.004307) | 0.088067 / 0.014526 (0.073541) | 0.103333 / 0.176557 (-0.073224) | 0.157061 / 0.737135 (-0.580074) | 0.105007 / 0.296338 (-0.191331) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.420826 / 0.215209 (0.205617) | 4.201656 / 2.077655 (2.124001) | 2.208336 / 1.504120 (0.704216) | 2.043780 / 1.541195 (0.502585) | 2.156215 / 1.468490 (0.687725) | 0.490485 / 4.584777 (-4.094292) | 3.611446 / 3.745712 (-0.134267) | 5.293140 / 5.269862 (0.023279) | 2.739778 / 4.565676 (-1.825899) | 0.058175 / 0.424275 (-0.366100) | 0.007633 / 0.007607 (0.000026) | 0.500773 / 0.226044 (0.274729) | 5.000900 / 2.268929 (2.731971) | 2.721200 / 55.444624 (-52.723424) | 2.349381 / 6.876477 (-4.527095) | 2.386261 / 2.142072 (0.244188) | 0.583174 / 4.805227 (-4.222053) | 0.134558 / 6.500664 (-6.366106) | 0.062157 / 0.075469 (-0.013312) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.351087 / 1.841788 (-0.490701) | 20.305703 / 8.074308 (12.231395) | 14.548518 / 10.191392 (4.357126) | 0.173720 / 0.680424 (-0.506704) | 0.018100 / 0.534201 (-0.516101) | 0.395187 / 0.579283 (-0.184097) | 0.414619 / 0.434364 (-0.019745) | 0.462515 / 0.540337 (-0.077823) | 0.617822 / 1.386936 (-0.769114) |

</details>
</details>

![](https://cml.dev/watermark.png#033d0a9de5c825fc9a6a9ce3c3d80eaab3493720 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006909 / 0.011353 (-0.004444) | 0.003954 / 0.011008 (-0.007054) | 0.084329 / 0.038508 (0.045821) | 0.074919 / 0.023109 (0.051809) | 0.319350 / 0.275898 (0.043451) | 0.347264 / 0.323480 (0.023785) | 0.005326 / 0.007986 (-0.002660) | 0.003323 / 0.004328 (-0.001006) | 0.064286 / 0.004250 (0.060036) | 0.054748 / 0.037052 (0.017696) | 0.324784 / 0.258489 (0.066295) | 0.361445 / 0.293841 (0.067605) | 0.031239 / 0.128546 (-0.097308) | 0.008361 / 0.075646 (-0.067286) | 0.287482 / 0.419271 (-0.131789) | 0.052093 / 0.043533 (0.008560) | 0.321454 / 0.255139 (0.066315) | 0.337999 / 0.283200 (0.054800) | 0.025807 / 0.141683 (-0.115876) | 1.501838 / 1.452155 (0.049683) | 1.574484 / 1.492716 (0.081767) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.193220 / 0.018006 (0.175214) | 0.448105 / 0.000490 (0.447615) | 0.002949 / 0.000200 (0.002749) | 0.000071 / 0.000054 (0.000016) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028517 / 0.037411 (-0.008894) | 0.087281 / 0.014526 (0.072755) | 0.098295 / 0.176557 (-0.078262) | 0.156972 / 0.737135 (-0.580163) | 0.101250 / 0.296338 (-0.195088) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.383734 / 0.215209 (0.168525) | 3.821293 / 2.077655 (1.743638) | 1.866487 / 1.504120 (0.362367) | 1.722195 / 1.541195 (0.181000) | 1.843762 / 1.468490 (0.375272) | 0.484813 / 4.584777 (-4.099964) | 3.535381 / 3.745712 (-0.210331) | 5.502338 / 5.269862 (0.232477) | 3.256078 / 4.565676 (-1.309599) | 0.057312 / 0.424275 (-0.366963) | 0.007305 / 0.007607 (-0.000302) | 0.461523 / 0.226044 (0.235479) | 4.611828 / 2.268929 (2.342899) | 2.337180 / 55.444624 (-53.107445) | 2.040956 / 6.876477 (-4.835521) | 2.241233 / 2.142072 (0.099160) | 0.583727 / 4.805227 (-4.221500) | 0.132427 / 6.500664 (-6.368237) | 0.060306 / 0.075469 (-0.015163) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.282223 / 1.841788 (-0.559565) | 19.439745 / 8.074308 (11.365437) | 13.627657 / 10.191392 (3.436265) | 0.158975 / 0.680424 (-0.521449) | 0.018599 / 0.534201 (-0.515601) | 0.391136 / 0.579283 (-0.188147) | 0.410947 / 0.434364 (-0.023417) | 0.453889 / 0.540337 (-0.086448) | 0.620928 / 1.386936 (-0.766008) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006428 / 0.011353 (-0.004925) | 0.003980 / 0.011008 (-0.007028) | 0.065006 / 0.038508 (0.026498) | 0.076541 / 0.023109 (0.053432) | 0.358518 / 0.275898 (0.082620) | 0.394397 / 0.323480 (0.070917) | 0.005845 / 0.007986 (-0.002140) | 0.003258 / 0.004328 (-0.001071) | 0.064436 / 0.004250 (0.060186) | 0.056691 / 0.037052 (0.019639) | 0.367369 / 0.258489 (0.108880) | 0.420345 / 0.293841 (0.126504) | 0.031047 / 0.128546 (-0.097499) | 0.008430 / 0.075646 (-0.067216) | 0.071280 / 0.419271 (-0.347991) | 0.048872 / 0.043533 (0.005339) | 0.360073 / 0.255139 (0.104934) | 0.384150 / 0.283200 (0.100951) | 0.023189 / 0.141683 (-0.118494) | 1.500251 / 1.452155 (0.048096) | 1.545910 / 1.492716 (0.053194) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.224861 / 0.018006 (0.206855) | 0.439901 / 0.000490 (0.439411) | 0.000372 / 0.000200 (0.000172) | 0.000054 / 0.000054 (-0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029914 / 0.037411 (-0.007497) | 0.086916 / 0.014526 (0.072390) | 0.099527 / 0.176557 (-0.077029) | 0.153031 / 0.737135 (-0.584104) | 0.100008 / 0.296338 (-0.196330) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.420305 / 0.215209 (0.205096) | 4.198224 / 2.077655 (2.120569) | 2.223807 / 1.504120 (0.719687) | 2.058475 / 1.541195 (0.517280) | 2.140405 / 1.468490 (0.671915) | 0.481224 / 4.584777 (-4.103553) | 3.593767 / 3.745712 (-0.151945) | 5.536710 / 5.269862 (0.266849) | 3.162048 / 4.565676 (-1.403629) | 0.056662 / 0.424275 (-0.367614) | 0.007301 / 0.007607 (-0.000306) | 0.507494 / 0.226044 (0.281450) | 5.047824 / 2.268929 (2.778896) | 2.715167 / 55.444624 (-52.729458) | 2.334916 / 6.876477 (-4.541560) | 2.406615 / 2.142072 (0.264543) | 0.572761 / 4.805227 (-4.232466) | 0.131248 / 6.500664 (-6.369416) | 0.062401 / 0.075469 (-0.013068) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.375896 / 1.841788 (-0.465892) | 19.836638 / 8.074308 (11.762329) | 14.246645 / 10.191392 (4.055253) | 0.164975 / 0.680424 (-0.515449) | 0.018293 / 0.534201 (-0.515908) | 0.394196 / 0.579283 (-0.185087) | 0.405895 / 0.434364 (-0.028469) | 0.459221 / 0.540337 (-0.081116) | 0.609898 / 1.386936 (-0.777038) |

</details>
</details>

![](https://cml.dev/watermark.png#f89210ad839c2225b64822dfa248f68ab29ad46f "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008463 / 0.011353 (-0.002890) | 0.004754 / 0.011008 (-0.006254) | 0.103574 / 0.038508 (0.065066) | 0.083541 / 0.023109 (0.060432) | 0.402498 / 0.275898 (0.126600) | 0.434944 / 0.323480 (0.111465) | 0.005766 / 0.007986 (-0.002219) | 0.003823 / 0.004328 (-0.000505) | 0.078433 / 0.004250 (0.074183) | 0.056948 / 0.037052 (0.019895) | 0.392539 / 0.258489 (0.134050) | 0.447226 / 0.293841 (0.153385) | 0.045845 / 0.128546 (-0.082701) | 0.014043 / 0.075646 (-0.061603) | 0.355768 / 0.419271 (-0.063503) | 0.065492 / 0.043533 (0.021960) | 0.408047 / 0.255139 (0.152908) | 0.468313 / 0.283200 (0.185113) | 0.033779 / 0.141683 (-0.107904) | 1.772198 / 1.452155 (0.320043) | 1.889127 / 1.492716 (0.396411) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.207107 / 0.018006 (0.189101) | 0.533261 / 0.000490 (0.532771) | 0.000864 / 0.000200 (0.000664) | 0.000105 / 0.000054 (0.000051) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032139 / 0.037411 (-0.005272) | 0.102002 / 0.014526 (0.087476) | 0.108780 / 0.176557 (-0.067777) | 0.202857 / 0.737135 (-0.534278) | 0.110378 / 0.296338 (-0.185960) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.582814 / 0.215209 (0.367605) | 5.870683 / 2.077655 (3.793028) | 2.510290 / 1.504120 (1.006171) | 2.146337 / 1.541195 (0.605142) | 2.239278 / 1.468490 (0.770788) | 0.861205 / 4.584777 (-3.723572) | 5.177394 / 3.745712 (1.431682) | 8.550713 / 5.269862 (3.280852) | 4.867715 / 4.565676 (0.302038) | 0.096665 / 0.424275 (-0.327610) | 0.008702 / 0.007607 (0.001095) | 0.748908 / 0.226044 (0.522863) | 7.302815 / 2.268929 (5.033887) | 3.205045 / 55.444624 (-52.239580) | 2.743914 / 6.876477 (-4.132562) | 2.831240 / 2.142072 (0.689167) | 1.103912 / 4.805227 (-3.701315) | 0.246075 / 6.500664 (-6.254589) | 0.092092 / 0.075469 (0.016623) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.591331 / 1.841788 (-0.250457) | 23.085848 / 8.074308 (15.011540) | 22.887963 / 10.191392 (12.696571) | 0.212735 / 0.680424 (-0.467689) | 0.027400 / 0.534201 (-0.506801) | 0.493822 / 0.579283 (-0.085461) | 0.574485 / 0.434364 (0.140121) | 0.574873 / 0.540337 (0.034536) | 0.826178 / 1.386936 (-0.560758) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009155 / 0.011353 (-0.002198) | 0.004976 / 0.011008 (-0.006032) | 0.079308 / 0.038508 (0.040799) | 0.093959 / 0.023109 (0.070850) | 0.449110 / 0.275898 (0.173212) | 0.493356 / 0.323480 (0.169876) | 0.006317 / 0.007986 (-0.001669) | 0.004179 / 0.004328 (-0.000150) | 0.076991 / 0.004250 (0.072740) | 0.061977 / 0.037052 (0.024924) | 0.493823 / 0.258489 (0.235333) | 0.491609 / 0.293841 (0.197768) | 0.049552 / 0.128546 (-0.078994) | 0.015174 / 0.075646 (-0.060472) | 0.090431 / 0.419271 (-0.328841) | 0.061597 / 0.043533 (0.018064) | 0.467672 / 0.255139 (0.212533) | 0.490542 / 0.283200 (0.207342) | 0.035048 / 0.141683 (-0.106635) | 1.807939 / 1.452155 (0.355784) | 1.854859 / 1.492716 (0.362142) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.236672 / 0.018006 (0.218666) | 0.542236 / 0.000490 (0.541746) | 0.016334 / 0.000200 (0.016134) | 0.000220 / 0.000054 (0.000165) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032051 / 0.037411 (-0.005360) | 0.115352 / 0.014526 (0.100826) | 0.125115 / 0.176557 (-0.051441) | 0.173670 / 0.737135 (-0.563466) | 0.117832 / 0.296338 (-0.178507) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.631513 / 0.215209 (0.416304) | 6.371688 / 2.077655 (4.294033) | 2.867240 / 1.504120 (1.363120) | 2.454907 / 1.541195 (0.913713) | 2.518860 / 1.468490 (1.050370) | 0.879973 / 4.584777 (-3.704804) | 5.170263 / 3.745712 (1.424551) | 7.986429 / 5.269862 (2.716567) | 4.828095 / 4.565676 (0.262418) | 0.097808 / 0.424275 (-0.326468) | 0.010541 / 0.007607 (0.002934) | 0.745601 / 0.226044 (0.519557) | 7.631683 / 2.268929 (5.362755) | 3.524255 / 55.444624 (-51.920369) | 2.866199 / 6.876477 (-4.010278) | 2.982483 / 2.142072 (0.840410) | 1.148957 / 4.805227 (-3.656270) | 0.217067 / 6.500664 (-6.283598) | 0.074357 / 0.075469 (-0.001112) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.714917 / 1.841788 (-0.126871) | 24.151348 / 8.074308 (16.077040) | 21.993604 / 10.191392 (11.802212) | 0.234883 / 0.680424 (-0.445541) | 0.028182 / 0.534201 (-0.506019) | 0.474050 / 0.579283 (-0.105233) | 0.557012 / 0.434364 (0.122648) | 0.537823 / 0.540337 (-0.002514) | 0.741488 / 1.386936 (-0.645448) |

</details>
</details>

![](https://cml.dev/watermark.png#c2e5a7a01a952a17d0424e93c3be2b4a5ffca7da "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007640 / 0.011353 (-0.003713) | 0.004776 / 0.011008 (-0.006232) | 0.101582 / 0.038508 (0.063074) | 0.085113 / 0.023109 (0.062003) | 0.376000 / 0.275898 (0.100102) | 0.421117 / 0.323480 (0.097637) | 0.006095 / 0.007986 (-0.001891) | 0.003884 / 0.004328 (-0.000445) | 0.077263 / 0.004250 (0.073013) | 0.065262 / 0.037052 (0.028210) | 0.384041 / 0.258489 (0.125552) | 0.442229 / 0.293841 (0.148388) | 0.035706 / 0.128546 (-0.092840) | 0.009996 / 0.075646 (-0.065651) | 0.344925 / 0.419271 (-0.074346) | 0.062358 / 0.043533 (0.018825) | 0.371738 / 0.255139 (0.116599) | 0.407093 / 0.283200 (0.123894) | 0.026996 / 0.141683 (-0.114687) | 1.762705 / 1.452155 (0.310550) | 1.846777 / 1.492716 (0.354061) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.219660 / 0.018006 (0.201653) | 0.521795 / 0.000490 (0.521305) | 0.005344 / 0.000200 (0.005145) | 0.000098 / 0.000054 (0.000044) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.036027 / 0.037411 (-0.001385) | 0.100309 / 0.014526 (0.085784) | 0.113041 / 0.176557 (-0.063515) | 0.190037 / 0.737135 (-0.547099) | 0.114552 / 0.296338 (-0.181786) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.466364 / 0.215209 (0.251154) | 4.638745 / 2.077655 (2.561090) | 2.317875 / 1.504120 (0.813755) | 2.099241 / 1.541195 (0.558046) | 2.149827 / 1.468490 (0.681337) | 0.578913 / 4.584777 (-4.005864) | 4.281866 / 3.745712 (0.536154) | 3.778453 / 5.269862 (-1.491408) | 2.411704 / 4.565676 (-2.153972) | 0.068556 / 0.424275 (-0.355719) | 0.008779 / 0.007607 (0.001172) | 0.553165 / 0.226044 (0.327121) | 5.524520 / 2.268929 (3.255591) | 2.848444 / 55.444624 (-52.596181) | 2.468591 / 6.876477 (-4.407885) | 2.652117 / 2.142072 (0.510045) | 0.694124 / 4.805227 (-4.111103) | 0.157087 / 6.500664 (-6.343577) | 0.070706 / 0.075469 (-0.004763) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.492031 / 1.841788 (-0.349757) | 23.086596 / 8.074308 (15.012288) | 16.791351 / 10.191392 (6.599959) | 0.203932 / 0.680424 (-0.476492) | 0.021736 / 0.534201 (-0.512464) | 0.468344 / 0.579283 (-0.110939) | 0.493790 / 0.434364 (0.059426) | 0.563226 / 0.540337 (0.022889) | 0.780384 / 1.386936 (-0.606553) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007980 / 0.011353 (-0.003373) | 0.004696 / 0.011008 (-0.006312) | 0.076712 / 0.038508 (0.038204) | 0.095915 / 0.023109 (0.072805) | 0.433615 / 0.275898 (0.157717) | 0.482477 / 0.323480 (0.158997) | 0.007029 / 0.007986 (-0.000957) | 0.003842 / 0.004328 (-0.000487) | 0.076331 / 0.004250 (0.072081) | 0.069755 / 0.037052 (0.032703) | 0.458914 / 0.258489 (0.200425) | 0.486155 / 0.293841 (0.192314) | 0.036966 / 0.128546 (-0.091580) | 0.010082 / 0.075646 (-0.065564) | 0.083886 / 0.419271 (-0.335385) | 0.059329 / 0.043533 (0.015796) | 0.453782 / 0.255139 (0.198643) | 0.459508 / 0.283200 (0.176308) | 0.028400 / 0.141683 (-0.113283) | 1.796406 / 1.452155 (0.344251) | 1.881161 / 1.492716 (0.388445) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.235053 / 0.018006 (0.217047) | 0.501907 / 0.000490 (0.501417) | 0.005211 / 0.000200 (0.005011) | 0.000101 / 0.000054 (0.000046) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.037752 / 0.037411 (0.000341) | 0.107299 / 0.014526 (0.092773) | 0.120307 / 0.176557 (-0.056250) | 0.187542 / 0.737135 (-0.549593) | 0.121805 / 0.296338 (-0.174533) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.490039 / 0.215209 (0.274830) | 4.919169 / 2.077655 (2.841515) | 2.520610 / 1.504120 (1.016490) | 2.324473 / 1.541195 (0.783279) | 2.421195 / 1.468490 (0.952705) | 0.576314 / 4.584777 (-4.008463) | 4.304752 / 3.745712 (0.559040) | 3.881151 / 5.269862 (-1.388710) | 2.409777 / 4.565676 (-2.155900) | 0.067400 / 0.424275 (-0.356875) | 0.009235 / 0.007607 (0.001627) | 0.586601 / 0.226044 (0.360556) | 5.850080 / 2.268929 (3.581152) | 3.064859 / 55.444624 (-52.379766) | 2.701734 / 6.876477 (-4.174743) | 2.926190 / 2.142072 (0.784117) | 0.698511 / 4.805227 (-4.106716) | 0.158273 / 6.500664 (-6.342392) | 0.074530 / 0.075469 (-0.000939) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.607113 / 1.841788 (-0.234674) | 23.499279 / 8.074308 (15.424971) | 17.049509 / 10.191392 (6.858117) | 0.175689 / 0.680424 (-0.504735) | 0.021762 / 0.534201 (-0.512439) | 0.491450 / 0.579283 (-0.087833) | 0.487557 / 0.434364 (0.053193) | 0.570104 / 0.540337 (0.029766) | 0.761527 / 1.386936 (-0.625409) |

</details>
</details>

![](https://cml.dev/watermark.png#7096c59e6a8f4d5b16f3b906075f9e2ed83bbb25 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008725 / 0.011353 (-0.002628) | 0.005156 / 0.011008 (-0.005852) | 0.095147 / 0.038508 (0.056639) | 0.084916 / 0.023109 (0.061807) | 0.390769 / 0.275898 (0.114871) | 0.434716 / 0.323480 (0.111237) | 0.005982 / 0.007986 (-0.002004) | 0.004323 / 0.004328 (-0.000006) | 0.074712 / 0.004250 (0.070461) | 0.058889 / 0.037052 (0.021837) | 0.403997 / 0.258489 (0.145508) | 0.443361 / 0.293841 (0.149520) | 0.045908 / 0.128546 (-0.082639) | 0.013562 / 0.075646 (-0.062085) | 0.330683 / 0.419271 (-0.088588) | 0.064821 / 0.043533 (0.021288) | 0.407202 / 0.255139 (0.152063) | 0.409930 / 0.283200 (0.126730) | 0.032693 / 0.141683 (-0.108990) | 1.630181 / 1.452155 (0.178026) | 1.729680 / 1.492716 (0.236963) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.261240 / 0.018006 (0.243234) | 0.581850 / 0.000490 (0.581360) | 0.002997 / 0.000200 (0.002797) | 0.000107 / 0.000054 (0.000053) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029279 / 0.037411 (-0.008133) | 0.085004 / 0.014526 (0.070478) | 0.127782 / 0.176557 (-0.048774) | 0.168852 / 0.737135 (-0.568283) | 0.098697 / 0.296338 (-0.197641) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.546417 / 0.215209 (0.331208) | 5.602186 / 2.077655 (3.524531) | 2.597049 / 1.504120 (1.092930) | 2.384880 / 1.541195 (0.843685) | 2.444516 / 1.468490 (0.976026) | 0.796562 / 4.584777 (-3.788214) | 5.239440 / 3.745712 (1.493727) | 7.087768 / 5.269862 (1.817906) | 4.308476 / 4.565676 (-0.257200) | 0.091215 / 0.424275 (-0.333060) | 0.007942 / 0.007607 (0.000335) | 0.690059 / 0.226044 (0.464015) | 6.727809 / 2.268929 (4.458880) | 3.294522 / 55.444624 (-52.150103) | 2.604088 / 6.876477 (-4.272389) | 2.786970 / 2.142072 (0.644898) | 0.918817 / 4.805227 (-3.886410) | 0.191451 / 6.500664 (-6.309213) | 0.069557 / 0.075469 (-0.005912) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.486377 / 1.841788 (-0.355411) | 22.363470 / 8.074308 (14.289162) | 19.963684 / 10.191392 (9.772292) | 0.204161 / 0.680424 (-0.476263) | 0.034570 / 0.534201 (-0.499631) | 0.467937 / 0.579283 (-0.111346) | 0.564870 / 0.434364 (0.130506) | 0.511133 / 0.540337 (-0.029204) | 0.777084 / 1.386936 (-0.609852) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008612 / 0.011353 (-0.002741) | 0.004993 / 0.011008 (-0.006015) | 0.080769 / 0.038508 (0.042261) | 0.075923 / 0.023109 (0.052814) | 0.442271 / 0.275898 (0.166373) | 0.495625 / 0.323480 (0.172146) | 0.006467 / 0.007986 (-0.001518) | 0.004001 / 0.004328 (-0.000328) | 0.077309 / 0.004250 (0.073059) | 0.063466 / 0.037052 (0.026414) | 0.452460 / 0.258489 (0.193971) | 0.494063 / 0.293841 (0.200223) | 0.045751 / 0.128546 (-0.082796) | 0.013402 / 0.075646 (-0.062245) | 0.085760 / 0.419271 (-0.333511) | 0.056532 / 0.043533 (0.012999) | 0.440596 / 0.255139 (0.185457) | 0.459540 / 0.283200 (0.176340) | 0.035897 / 0.141683 (-0.105786) | 1.728264 / 1.452155 (0.276109) | 1.808142 / 1.492716 (0.315426) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.285094 / 0.018006 (0.267088) | 0.598440 / 0.000490 (0.597950) | 0.003476 / 0.000200 (0.003276) | 0.000103 / 0.000054 (0.000048) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035106 / 0.037411 (-0.002305) | 0.091724 / 0.014526 (0.077198) | 0.122803 / 0.176557 (-0.053754) | 0.182114 / 0.737135 (-0.555022) | 0.116196 / 0.296338 (-0.180143) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.585420 / 0.215209 (0.370211) | 5.790370 / 2.077655 (3.712715) | 2.833247 / 1.504120 (1.329127) | 2.627949 / 1.541195 (1.086755) | 2.643050 / 1.468490 (1.174560) | 0.792036 / 4.584777 (-3.792741) | 5.145084 / 3.745712 (1.399372) | 4.423679 / 5.269862 (-0.846182) | 2.802778 / 4.565676 (-1.762898) | 0.093983 / 0.424275 (-0.330292) | 0.009260 / 0.007607 (0.001652) | 0.720302 / 0.226044 (0.494258) | 7.116959 / 2.268929 (4.848031) | 3.574782 / 55.444624 (-51.869843) | 3.009330 / 6.876477 (-3.867147) | 3.126488 / 2.142072 (0.984415) | 0.949144 / 4.805227 (-3.856083) | 0.195143 / 6.500664 (-6.305521) | 0.072490 / 0.075469 (-0.002979) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.626368 / 1.841788 (-0.215419) | 23.683021 / 8.074308 (15.608713) | 20.085297 / 10.191392 (9.893905) | 0.267057 / 0.680424 (-0.413367) | 0.028306 / 0.534201 (-0.505894) | 0.478448 / 0.579283 (-0.100835) | 0.597619 / 0.434364 (0.163256) | 0.544737 / 0.540337 (0.004399) | 0.761805 / 1.386936 (-0.625131) |

</details>
</details>

![](https://cml.dev/watermark.png#7ac53b590916c8d859fabcc2ef23c12add7f22f7 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009359 / 0.011353 (-0.001994) | 0.004848 / 0.011008 (-0.006160) | 0.099471 / 0.038508 (0.060963) | 0.079483 / 0.023109 (0.056373) | 0.375281 / 0.275898 (0.099383) | 0.415566 / 0.323480 (0.092086) | 0.006317 / 0.007986 (-0.001669) | 0.005145 / 0.004328 (0.000817) | 0.080345 / 0.004250 (0.076094) | 0.064540 / 0.037052 (0.027487) | 0.385897 / 0.258489 (0.127408) | 0.432576 / 0.293841 (0.138735) | 0.055109 / 0.128546 (-0.073437) | 0.014166 / 0.075646 (-0.061480) | 0.350870 / 0.419271 (-0.068402) | 0.087483 / 0.043533 (0.043950) | 0.402288 / 0.255139 (0.147149) | 0.391997 / 0.283200 (0.108798) | 0.045233 / 0.141683 (-0.096450) | 1.795002 / 1.452155 (0.342847) | 1.839063 / 1.492716 (0.346347) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.220851 / 0.018006 (0.202845) | 0.513391 / 0.000490 (0.512901) | 0.003740 / 0.000200 (0.003540) | 0.000107 / 0.000054 (0.000053) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035287 / 0.037411 (-0.002124) | 0.090670 / 0.014526 (0.076144) | 0.115651 / 0.176557 (-0.060905) | 0.180469 / 0.737135 (-0.556667) | 0.106955 / 0.296338 (-0.189384) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.632381 / 0.215209 (0.417172) | 6.185151 / 2.077655 (4.107497) | 2.548263 / 1.504120 (1.044143) | 2.194931 / 1.541195 (0.653737) | 2.368685 / 1.468490 (0.900194) | 0.956467 / 4.584777 (-3.628310) | 5.280904 / 3.745712 (1.535192) | 4.783057 / 5.269862 (-0.486805) | 3.218493 / 4.565676 (-1.347184) | 0.103545 / 0.424275 (-0.320730) | 0.008424 / 0.007607 (0.000817) | 0.736303 / 0.226044 (0.510259) | 7.354305 / 2.268929 (5.085376) | 3.280670 / 55.444624 (-52.163954) | 2.478628 / 6.876477 (-4.397848) | 2.623290 / 2.142072 (0.481217) | 1.033064 / 4.805227 (-3.772163) | 0.206496 / 6.500664 (-6.294168) | 0.066449 / 0.075469 (-0.009020) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.508756 / 1.841788 (-0.333031) | 21.866012 / 8.074308 (13.791704) | 21.887761 / 10.191392 (11.696369) | 0.231415 / 0.680424 (-0.449008) | 0.028917 / 0.534201 (-0.505284) | 0.468761 / 0.579283 (-0.110522) | 0.568236 / 0.434364 (0.133872) | 0.550156 / 0.540337 (0.009818) | 0.783197 / 1.386936 (-0.603739) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009413 / 0.011353 (-0.001939) | 0.004951 / 0.011008 (-0.006058) | 0.071402 / 0.038508 (0.032893) | 0.068455 / 0.023109 (0.045346) | 0.425216 / 0.275898 (0.149318) | 0.431928 / 0.323480 (0.108448) | 0.006477 / 0.007986 (-0.001509) | 0.003891 / 0.004328 (-0.000437) | 0.076898 / 0.004250 (0.072647) | 0.057522 / 0.037052 (0.020470) | 0.449585 / 0.258489 (0.191096) | 0.431356 / 0.293841 (0.137515) | 0.049728 / 0.128546 (-0.078818) | 0.014456 / 0.075646 (-0.061190) | 0.084618 / 0.419271 (-0.334653) | 0.064482 / 0.043533 (0.020949) | 0.456377 / 0.255139 (0.201238) | 0.433949 / 0.283200 (0.150749) | 0.036577 / 0.141683 (-0.105106) | 1.819742 / 1.452155 (0.367588) | 1.694691 / 1.492716 (0.201975) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.224610 / 0.018006 (0.206604) | 0.494586 / 0.000490 (0.494096) | 0.004506 / 0.000200 (0.004307) | 0.000119 / 0.000054 (0.000065) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033172 / 0.037411 (-0.004239) | 0.100562 / 0.014526 (0.086036) | 0.116499 / 0.176557 (-0.060058) | 0.153717 / 0.737135 (-0.583418) | 0.140047 / 0.296338 (-0.156291) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.635922 / 0.215209 (0.420713) | 6.359792 / 2.077655 (4.282137) | 2.689083 / 1.504120 (1.184963) | 2.330574 / 1.541195 (0.789380) | 2.583535 / 1.468490 (1.115044) | 0.902737 / 4.584777 (-3.682040) | 5.136586 / 3.745712 (1.390874) | 4.570824 / 5.269862 (-0.699037) | 3.029953 / 4.565676 (-1.535724) | 0.103961 / 0.424275 (-0.320314) | 0.007908 / 0.007607 (0.000301) | 0.723290 / 0.226044 (0.497246) | 7.678599 / 2.268929 (5.409671) | 3.342522 / 55.444624 (-52.102102) | 2.774659 / 6.876477 (-4.101817) | 2.966496 / 2.142072 (0.824423) | 1.025395 / 4.805227 (-3.779832) | 0.222246 / 6.500664 (-6.278418) | 0.072455 / 0.075469 (-0.003014) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.603637 / 1.841788 (-0.238151) | 21.387722 / 8.074308 (13.313414) | 22.855221 / 10.191392 (12.663829) | 0.222147 / 0.680424 (-0.458277) | 0.030763 / 0.534201 (-0.503438) | 0.472586 / 0.579283 (-0.106697) | 0.560161 / 0.434364 (0.125797) | 0.551941 / 0.540337 (0.011604) | 0.711254 / 1.386936 (-0.675682) |

</details>
</details>

![](https://cml.dev/watermark.png#85cf123e553ff282b43ad1d1877ba2c40d206d52 "CML watermark")