Issue: Fix Parquet loading with `columns`
State: closed
Author: mariosasko

Fix #6149 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008368 / 0.011353 (-0.002985) | 0.004754 / 0.011008 (-0.006254) | 0.096646 / 0.038508 (0.058138) | 0.088980 / 0.023109 (0.065871) | 0.374532 / 0.275898 (0.098633) | 0.404840 / 0.323480 (0.081360) | 0.006026 / 0.007986 (-0.001960) | 0.005716 / 0.004328 (0.001387) | 0.076297 / 0.004250 (0.072047) | 0.072335 / 0.037052 (0.035283) | 0.379435 / 0.258489 (0.120946) | 0.423449 / 0.293841 (0.129608) | 0.041344 / 0.128546 (-0.087202) | 0.009758 / 0.075646 (-0.065889) | 0.341550 / 0.419271 (-0.077721) | 0.068559 / 0.043533 (0.025026) | 0.368313 / 0.255139 (0.113174) | 0.415147 / 0.283200 (0.131947) | 0.028692 / 0.141683 (-0.112990) | 1.816198 / 1.452155 (0.364044) | 1.983351 / 1.492716 (0.490635) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.222712 / 0.018006 (0.204706) | 0.517850 / 0.000490 (0.517360) | 0.004436 / 0.000200 (0.004236) | 0.000094 / 0.000054 (0.000040) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033168 / 0.037411 (-0.004243) | 0.101353 / 0.014526 (0.086827) | 0.113235 / 0.176557 (-0.063322) | 0.180308 / 0.737135 (-0.556827) | 0.114604 / 0.296338 (-0.181734) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.454415 / 0.215209 (0.239206) | 4.500355 / 2.077655 (2.422701) | 2.188223 / 1.504120 (0.684103) | 1.974256 / 1.541195 (0.433061) | 2.067331 / 1.468490 (0.598841) | 0.572982 / 4.584777 (-4.011795) | 4.239160 / 3.745712 (0.493448) | 3.836812 / 5.269862 (-1.433049) | 2.367022 / 4.565676 (-2.198655) | 0.066886 / 0.424275 (-0.357389) | 0.009111 / 0.007607 (0.001504) | 0.539881 / 0.226044 (0.313837) | 5.362247 / 2.268929 (3.093319) | 2.784044 / 55.444624 (-52.660580) | 2.320975 / 6.876477 (-4.555502) | 2.543108 / 2.142072 (0.401036) | 0.685751 / 4.805227 (-4.119477) | 0.156840 / 6.500664 (-6.343824) | 0.071764 / 0.075469 (-0.003705) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.549830 / 1.841788 (-0.291958) | 22.799622 / 8.074308 (14.725314) | 16.750692 / 10.191392 (6.559300) | 0.196192 / 0.680424 (-0.484232) | 0.024518 / 0.534201 (-0.509683) | 0.479302 / 0.579283 (-0.099981) | 0.522256 / 0.434364 (0.087892) | 0.545809 / 0.540337 (0.005471) | 0.748437 / 1.386936 (-0.638499) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007789 / 0.011353 (-0.003564) | 0.004563 / 0.011008 (-0.006445) | 0.074631 / 0.038508 (0.036123) | 0.086892 / 0.023109 (0.063783) | 0.427014 / 0.275898 (0.151116) | 0.463257 / 0.323480 (0.139777) | 0.005987 / 0.007986 (-0.001999) | 0.003803 / 0.004328 (-0.000526) | 0.074799 / 0.004250 (0.070549) | 0.063473 / 0.037052 (0.026420) | 0.429905 / 0.258489 (0.171416) | 0.468967 / 0.293841 (0.175127) | 0.036768 / 0.128546 (-0.091778) | 0.009675 / 0.075646 (-0.065971) | 0.082546 / 0.419271 (-0.336725) | 0.058027 / 0.043533 (0.014494) | 0.429813 / 0.255139 (0.174674) | 0.449200 / 0.283200 (0.166001) | 0.026713 / 0.141683 (-0.114969) | 1.812022 / 1.452155 (0.359867) | 1.847305 / 1.492716 (0.354589) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.320383 / 0.018006 (0.302377) | 0.485995 / 0.000490 (0.485505) | 0.024365 / 0.000200 (0.024165) | 0.000156 / 0.000054 (0.000101) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.036341 / 0.037411 (-0.001071) | 0.104635 / 0.014526 (0.090110) | 0.119456 / 0.176557 (-0.057101) | 0.182042 / 0.737135 (-0.555093) | 0.118944 / 0.296338 (-0.177395) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.506410 / 0.215209 (0.291201) | 5.061119 / 2.077655 (2.983465) | 2.756557 / 1.504120 (1.252437) | 2.546504 / 1.541195 (1.005309) | 2.585509 / 1.468490 (1.117019) | 0.564291 / 4.584777 (-4.020486) | 4.281219 / 3.745712 (0.535507) | 3.919439 / 5.269862 (-1.350423) | 2.588788 / 4.565676 (-1.976889) | 0.066900 / 0.424275 (-0.357375) | 0.008680 / 0.007607 (0.001073) | 0.598435 / 0.226044 (0.372390) | 5.976054 / 2.268929 (3.707125) | 3.260211 / 55.444624 (-52.184414) | 2.874597 / 6.876477 (-4.001880) | 3.105769 / 2.142072 (0.963697) | 0.692938 / 4.805227 (-4.112289) | 0.157777 / 6.500664 (-6.342887) | 0.073128 / 0.075469 (-0.002341) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.559380 / 1.841788 (-0.282408) | 22.986540 / 8.074308 (14.912232) | 16.305564 / 10.191392 (6.114172) | 0.174939 / 0.680424 (-0.505485) | 0.021932 / 0.534201 (-0.512269) | 0.468162 / 0.579283 (-0.111121) | 0.472610 / 0.434364 (0.038246) | 0.574574 / 0.540337 (0.034237) | 0.783505 / 1.386936 (-0.603431) |

</details>
</details>

![](https://cml.dev/watermark.png#550923b5d6ae64eb20b8f66da843395e9fa404ac "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.012553 / 0.011353 (0.001201) | 0.005358 / 0.011008 (-0.005650) | 0.108338 / 0.038508 (0.069830) | 0.101105 / 0.023109 (0.077995) | 0.416808 / 0.275898 (0.140910) | 0.454599 / 0.323480 (0.131119) | 0.006665 / 0.007986 (-0.001321) | 0.004186 / 0.004328 (-0.000143) | 0.084900 / 0.004250 (0.080649) | 0.062881 / 0.037052 (0.025829) | 0.424423 / 0.258489 (0.165934) | 0.482651 / 0.293841 (0.188810) | 0.055740 / 0.128546 (-0.072807) | 0.014469 / 0.075646 (-0.061177) | 0.383267 / 0.419271 (-0.036005) | 0.067487 / 0.043533 (0.023955) | 0.414983 / 0.255139 (0.159844) | 0.459437 / 0.283200 (0.176237) | 0.038679 / 0.141683 (-0.103004) | 1.828002 / 1.452155 (0.375847) | 1.951946 / 1.492716 (0.459230) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.288033 / 0.018006 (0.270027) | 0.603536 / 0.000490 (0.603046) | 0.004874 / 0.000200 (0.004674) | 0.000138 / 0.000054 (0.000084) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031988 / 0.037411 (-0.005423) | 0.095807 / 0.014526 (0.081281) | 0.113459 / 0.176557 (-0.063098) | 0.182012 / 0.737135 (-0.555123) | 0.113121 / 0.296338 (-0.183217) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.620709 / 0.215209 (0.405500) | 6.096569 / 2.077655 (4.018915) | 2.754612 / 1.504120 (1.250492) | 2.449786 / 1.541195 (0.908591) | 2.470694 / 1.468490 (1.002204) | 0.837016 / 4.584777 (-3.747761) | 5.237290 / 3.745712 (1.491578) | 4.713220 / 5.269862 (-0.556642) | 3.020934 / 4.565676 (-1.544743) | 0.096892 / 0.424275 (-0.327383) | 0.009423 / 0.007607 (0.001816) | 0.720313 / 0.226044 (0.494269) | 7.369673 / 2.268929 (5.100744) | 3.550384 / 55.444624 (-51.894241) | 2.868868 / 6.876477 (-4.007609) | 3.081469 / 2.142072 (0.939397) | 1.042968 / 4.805227 (-3.762259) | 0.232530 / 6.500664 (-6.268134) | 0.080805 / 0.075469 (0.005336) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.645777 / 1.841788 (-0.196011) | 24.590862 / 8.074308 (16.516554) | 21.315496 / 10.191392 (11.124104) | 0.228796 / 0.680424 (-0.451628) | 0.028479 / 0.534201 (-0.505722) | 0.494413 / 0.579283 (-0.084870) | 0.582773 / 0.434364 (0.148409) | 0.552575 / 0.540337 (0.012238) | 0.787217 / 1.386936 (-0.599719) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008743 / 0.011353 (-0.002609) | 0.005253 / 0.011008 (-0.005755) | 0.083766 / 0.038508 (0.045257) | 0.086305 / 0.023109 (0.063195) | 0.520171 / 0.275898 (0.244273) | 0.565812 / 0.323480 (0.242332) | 0.006465 / 0.007986 (-0.001520) | 0.004585 / 0.004328 (0.000257) | 0.085344 / 0.004250 (0.081094) | 0.063418 / 0.037052 (0.026366) | 0.519759 / 0.258489 (0.261270) | 0.552770 / 0.293841 (0.258929) | 0.049439 / 0.128546 (-0.079107) | 0.017564 / 0.075646 (-0.058082) | 0.092713 / 0.419271 (-0.326559) | 0.065837 / 0.043533 (0.022305) | 0.516133 / 0.255139 (0.260994) | 0.539813 / 0.283200 (0.256613) | 0.036531 / 0.141683 (-0.105152) | 1.919275 / 1.452155 (0.467121) | 2.039987 / 1.492716 (0.547271) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.297978 / 0.018006 (0.279972) | 0.608243 / 0.000490 (0.607753) | 0.006611 / 0.000200 (0.006411) | 0.000117 / 0.000054 (0.000062) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033909 / 0.037411 (-0.003503) | 0.106370 / 0.014526 (0.091844) | 0.119032 / 0.176557 (-0.057524) | 0.180319 / 0.737135 (-0.556816) | 0.122826 / 0.296338 (-0.173513) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.639265 / 0.215209 (0.424056) | 6.248430 / 2.077655 (4.170775) | 2.944760 / 1.504120 (1.440640) | 2.654005 / 1.541195 (1.112811) | 2.733625 / 1.468490 (1.265134) | 0.837172 / 4.584777 (-3.747605) | 5.245084 / 3.745712 (1.499372) | 4.722614 / 5.269862 (-0.547248) | 3.008286 / 4.565676 (-1.557391) | 0.102340 / 0.424275 (-0.321935) | 0.009433 / 0.007607 (0.001826) | 0.762991 / 0.226044 (0.536946) | 7.385020 / 2.268929 (5.116092) | 3.787648 / 55.444624 (-51.656977) | 3.234345 / 6.876477 (-3.642132) | 3.394444 / 2.142072 (1.252371) | 1.023472 / 4.805227 (-3.781756) | 0.208199 / 6.500664 (-6.292465) | 0.081513 / 0.075469 (0.006043) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.795864 / 1.841788 (-0.045923) | 25.270852 / 8.074308 (17.196544) | 23.356413 / 10.191392 (13.165021) | 0.228002 / 0.680424 (-0.452422) | 0.031851 / 0.534201 (-0.502350) | 0.499424 / 0.579283 (-0.079859) | 0.588027 / 0.434364 (0.153664) | 0.581746 / 0.540337 (0.041408) | 0.814183 / 1.386936 (-0.572753) |

</details>
</details>

![](https://cml.dev/watermark.png#33ee536876a667403ee44574bd685073261c4903 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006477 / 0.011353 (-0.004876) | 0.003878 / 0.011008 (-0.007130) | 0.084085 / 0.038508 (0.045577) | 0.071297 / 0.023109 (0.048188) | 0.309176 / 0.275898 (0.033278) | 0.342830 / 0.323480 (0.019350) | 0.005189 / 0.007986 (-0.002796) | 0.003263 / 0.004328 (-0.001065) | 0.063920 / 0.004250 (0.059670) | 0.052233 / 0.037052 (0.015180) | 0.324830 / 0.258489 (0.066341) | 0.357956 / 0.293841 (0.064115) | 0.030459 / 0.128546 (-0.098087) | 0.008350 / 0.075646 (-0.067297) | 0.287330 / 0.419271 (-0.131942) | 0.051005 / 0.043533 (0.007473) | 0.309227 / 0.255139 (0.054088) | 0.346184 / 0.283200 (0.062984) | 0.023961 / 0.141683 (-0.117722) | 1.463983 / 1.452155 (0.011829) | 1.573036 / 1.492716 (0.080319) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.205653 / 0.018006 (0.187647) | 0.457336 / 0.000490 (0.456846) | 0.005347 / 0.000200 (0.005147) | 0.000079 / 0.000054 (0.000025) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028080 / 0.037411 (-0.009332) | 0.081755 / 0.014526 (0.067229) | 0.095716 / 0.176557 (-0.080841) | 0.151340 / 0.737135 (-0.585795) | 0.097174 / 0.296338 (-0.199164) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.390725 / 0.215209 (0.175516) | 3.899114 / 2.077655 (1.821459) | 1.895352 / 1.504120 (0.391232) | 1.716072 / 1.541195 (0.174877) | 1.784952 / 1.468490 (0.316462) | 0.477247 / 4.584777 (-4.107530) | 3.606641 / 3.745712 (-0.139071) | 3.203337 / 5.269862 (-2.066524) | 2.017003 / 4.565676 (-2.548674) | 0.056182 / 0.424275 (-0.368094) | 0.007508 / 0.007607 (-0.000099) | 0.461965 / 0.226044 (0.235921) | 4.605926 / 2.268929 (2.336997) | 2.466695 / 55.444624 (-52.977929) | 2.136376 / 6.876477 (-4.740100) | 2.277334 / 2.142072 (0.135261) | 0.576119 / 4.805227 (-4.229109) | 0.131497 / 6.500664 (-6.369167) | 0.060068 / 0.075469 (-0.015401) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.262681 / 1.841788 (-0.579107) | 19.411572 / 8.074308 (11.337264) | 14.383421 / 10.191392 (4.192029) | 0.166115 / 0.680424 (-0.514308) | 0.018366 / 0.534201 (-0.515835) | 0.393903 / 0.579283 (-0.185380) | 0.408788 / 0.434364 (-0.025576) | 0.461796 / 0.540337 (-0.078541) | 0.628460 / 1.386936 (-0.758476) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006501 / 0.011353 (-0.004852) | 0.003915 / 0.011008 (-0.007093) | 0.065245 / 0.038508 (0.026737) | 0.073146 / 0.023109 (0.050037) | 0.363537 / 0.275898 (0.087639) | 0.391571 / 0.323480 (0.068092) | 0.005181 / 0.007986 (-0.002805) | 0.003272 / 0.004328 (-0.001056) | 0.065060 / 0.004250 (0.060810) | 0.054302 / 0.037052 (0.017249) | 0.361571 / 0.258489 (0.103082) | 0.400221 / 0.293841 (0.106380) | 0.030762 / 0.128546 (-0.097784) | 0.008449 / 0.075646 (-0.067197) | 0.071148 / 0.419271 (-0.348123) | 0.048111 / 0.043533 (0.004578) | 0.360327 / 0.255139 (0.105188) | 0.379073 / 0.283200 (0.095874) | 0.024367 / 0.141683 (-0.117316) | 1.451080 / 1.452155 (-0.001074) | 1.510818 / 1.492716 (0.018102) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.267078 / 0.018006 (0.249072) | 0.454074 / 0.000490 (0.453584) | 0.015055 / 0.000200 (0.014855) | 0.000129 / 0.000054 (0.000075) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030916 / 0.037411 (-0.006496) | 0.089212 / 0.014526 (0.074686) | 0.100005 / 0.176557 (-0.076552) | 0.155100 / 0.737135 (-0.582035) | 0.101759 / 0.296338 (-0.194580) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.412826 / 0.215209 (0.197616) | 4.122520 / 2.077655 (2.044865) | 2.107870 / 1.504120 (0.603750) | 1.911936 / 1.541195 (0.370741) | 1.984936 / 1.468490 (0.516446) | 0.483835 / 4.584777 (-4.100942) | 3.641860 / 3.745712 (-0.103852) | 3.220540 / 5.269862 (-2.049322) | 2.015521 / 4.565676 (-2.550155) | 0.056913 / 0.424275 (-0.367362) | 0.007285 / 0.007607 (-0.000322) | 0.484886 / 0.226044 (0.258842) | 4.854734 / 2.268929 (2.585805) | 2.593550 / 55.444624 (-52.851074) | 2.233904 / 6.876477 (-4.642572) | 2.438858 / 2.142072 (0.296785) | 0.580880 / 4.805227 (-4.224347) | 0.133891 / 6.500664 (-6.366773) | 0.061678 / 0.075469 (-0.013791) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.336843 / 1.841788 (-0.504944) | 19.731571 / 8.074308 (11.657263) | 14.290228 / 10.191392 (4.098836) | 0.167635 / 0.680424 (-0.512789) | 0.018767 / 0.534201 (-0.515434) | 0.394953 / 0.579283 (-0.184330) | 0.407711 / 0.434364 (-0.026653) | 0.472371 / 0.540337 (-0.067966) | 0.655278 / 1.386936 (-0.731658) |

</details>
</details>

![](https://cml.dev/watermark.png#528b15f775a4724836bdefdc38d932c06484d702 "CML watermark")