Release: 2.14.3
Opened by albertvillanova (closed)


Comments:
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007578 / 0.011353 (-0.003775) | 0.004271 / 0.011008 (-0.006738) | 0.086607 / 0.038508 (0.048098) | 0.063209 / 0.023109 (0.040099) | 0.351724 / 0.275898 (0.075826) | 0.399261 / 0.323480 (0.075781) | 0.004767 / 0.007986 (-0.003219) | 0.003487 / 0.004328 (-0.000842) | 0.071483 / 0.004250 (0.067233) | 0.051281 / 0.037052 (0.014229) | 0.387726 / 0.258489 (0.129237) | 0.408446 / 0.293841 (0.114605) | 0.041189 / 0.128546 (-0.087357) | 0.012446 / 0.075646 (-0.063200) | 0.331147 / 0.419271 (-0.088124) | 0.056721 / 0.043533 (0.013188) | 0.361306 / 0.255139 (0.106167) | 0.409651 / 0.283200 (0.126451) | 0.035485 / 0.141683 (-0.106198) | 1.461391 / 1.452155 (0.009236) | 1.554820 / 1.492716 (0.062104) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.237119 / 0.018006 (0.219113) | 0.518731 / 0.000490 (0.518241) | 0.004192 / 0.000200 (0.003992) | 0.000114 / 0.000054 (0.000059) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024912 / 0.037411 (-0.012499) | 0.089420 / 0.014526 (0.074894) | 0.091209 / 0.176557 (-0.085347) | 0.152580 / 0.737135 (-0.584555) | 0.089660 / 0.296338 (-0.206678) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.515223 / 0.215209 (0.300014) | 5.328359 / 2.077655 (3.250705) | 1.974326 / 1.504120 (0.470206) | 1.665216 / 1.541195 (0.124021) | 1.736040 / 1.468490 (0.267550) | 0.734746 / 4.584777 (-3.850031) | 4.186613 / 3.745712 (0.440901) | 3.535760 / 5.269862 (-1.734102) | 2.333247 / 4.565676 (-2.232429) | 0.071845 / 0.424275 (-0.352430) | 0.006147 / 0.007607 (-0.001460) | 0.546649 / 0.226044 (0.320605) | 5.452281 / 2.268929 (3.183353) | 2.512984 / 55.444624 (-52.931640) | 2.104210 / 6.876477 (-4.772267) | 2.409251 / 2.142072 (0.267178) | 0.822797 / 4.805227 (-3.982430) | 0.166648 / 6.500664 (-6.334016) | 0.056350 / 0.075469 (-0.019119) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.397798 / 1.841788 (-0.443989) | 20.549399 / 8.074308 (12.475091) | 19.118168 / 10.191392 (8.926776) | 0.216361 / 0.680424 (-0.464063) | 0.027064 / 0.534201 (-0.507136) | 0.410762 / 0.579283 (-0.168521) | 0.559225 / 0.434364 (0.124861) | 0.468028 / 0.540337 (-0.072309) | 0.691520 / 1.386936 (-0.695416) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006463 / 0.011353 (-0.004890) | 0.003879 / 0.011008 (-0.007130) | 0.058723 / 0.038508 (0.020215) | 0.057202 / 0.023109 (0.034092) | 0.344397 / 0.275898 (0.068499) | 0.360388 / 0.323480 (0.036908) | 0.005502 / 0.007986 (-0.002483) | 0.004101 / 0.004328 (-0.000227) | 0.058168 / 0.004250 (0.053917) | 0.059112 / 0.037052 (0.022060) | 0.362206 / 0.258489 (0.103717) | 0.386444 / 0.293841 (0.092603) | 0.036613 / 0.128546 (-0.091934) | 0.010482 / 0.075646 (-0.065165) | 0.065850 / 0.419271 (-0.353421) | 0.046528 / 0.043533 (0.002995) | 0.349568 / 0.255139 (0.094429) | 0.360181 / 0.283200 (0.076981) | 0.029030 / 0.141683 (-0.112653) | 1.314569 / 1.452155 (-0.137586) | 1.422393 / 1.492716 (-0.070324) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.281554 / 0.018006 (0.263548) | 0.608018 / 0.000490 (0.607528) | 0.004568 / 0.000200 (0.004368) | 0.000182 / 0.000054 (0.000127) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023515 / 0.037411 (-0.013896) | 0.072994 / 0.014526 (0.058468) | 0.080688 / 0.176557 (-0.095868) | 0.125904 / 0.737135 (-0.611232) | 0.085457 / 0.296338 (-0.210882) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.471530 / 0.215209 (0.256321) | 4.796197 / 2.077655 (2.718542) | 2.189181 / 1.504120 (0.685061) | 1.886649 / 1.541195 (0.345454) | 1.871067 / 1.468490 (0.402577) | 0.661043 / 4.584777 (-3.923734) | 4.344027 / 3.745712 (0.598315) | 3.656967 / 5.269862 (-1.612895) | 2.286033 / 4.565676 (-2.279644) | 0.079146 / 0.424275 (-0.345129) | 0.006840 / 0.007607 (-0.000767) | 0.588750 / 0.226044 (0.362706) | 6.301286 / 2.268929 (4.032357) | 3.074702 / 55.444624 (-52.369923) | 2.398739 / 6.876477 (-4.477738) | 2.555057 / 2.142072 (0.412985) | 0.874189 / 4.805227 (-3.931038) | 0.191423 / 6.500664 (-6.309241) | 0.061227 / 0.075469 (-0.014242) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.472763 / 1.841788 (-0.369024) | 19.441304 / 8.074308 (11.366996) | 15.974276 / 10.191392 (5.782884) | 0.172503 / 0.680424 (-0.507921) | 0.027016 / 0.534201 (-0.507185) | 0.356085 / 0.579283 (-0.223198) | 0.473251 / 0.434364 (0.038887) | 0.427949 / 0.540337 (-0.112388) | 0.588924 / 1.386936 (-0.798013) |

</details>
</details>

![](https://cml.dev/watermark.png#0973da6e60ac7c1d24229ba6aa6881747b21858a "CML watermark")


_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006166 / 0.011353 (-0.005187) | 0.003558 / 0.011008 (-0.007450) | 0.080576 / 0.038508 (0.042068) | 0.066542 / 0.023109 (0.043432) | 0.323997 / 0.275898 (0.048099) | 0.369828 / 0.323480 (0.046348) | 0.004896 / 0.007986 (-0.003090) | 0.002909 / 0.004328 (-0.001419) | 0.062553 / 0.004250 (0.058302) | 0.049795 / 0.037052 (0.012742) | 0.321369 / 0.258489 (0.062880) | 0.422860 / 0.293841 (0.129019) | 0.027394 / 0.128546 (-0.101152) | 0.007954 / 0.075646 (-0.067693) | 0.264122 / 0.419271 (-0.155149) | 0.044881 / 0.043533 (0.001349) | 0.316702 / 0.255139 (0.061563) | 0.374718 / 0.283200 (0.091518) | 0.021728 / 0.141683 (-0.119955) | 1.394456 / 1.452155 (-0.057699) | 1.474936 / 1.492716 (-0.017780) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.191902 / 0.018006 (0.173896) | 0.430468 / 0.000490 (0.429979) | 0.003790 / 0.000200 (0.003590) | 0.000069 / 0.000054 (0.000015) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024974 / 0.037411 (-0.012438) | 0.073053 / 0.014526 (0.058527) | 0.083801 / 0.176557 (-0.092756) | 0.143457 / 0.737135 (-0.593678) | 0.085099 / 0.296338 (-0.211240) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.428411 / 0.215209 (0.213202) | 4.278077 / 2.077655 (2.200422) | 2.230039 / 1.504120 (0.725919) | 2.057191 / 1.541195 (0.515996) | 2.120109 / 1.468490 (0.651619) | 0.495242 / 4.584777 (-4.089535) | 3.031299 / 3.745712 (-0.714413) | 2.802685 / 5.269862 (-2.467176) | 1.839828 / 4.565676 (-2.725849) | 0.056875 / 0.424275 (-0.367401) | 0.006446 / 0.007607 (-0.001161) | 0.498958 / 0.226044 (0.272913) | 4.980440 / 2.268929 (2.711511) | 2.659659 / 55.444624 (-52.784965) | 2.315174 / 6.876477 (-4.561303) | 2.475920 / 2.142072 (0.333848) | 0.586946 / 4.805227 (-4.218282) | 0.124291 / 6.500664 (-6.376373) | 0.060701 / 0.075469 (-0.014768) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.245062 / 1.841788 (-0.596725) | 18.201444 / 8.074308 (10.127136) | 13.723271 / 10.191392 (3.531879) | 0.130203 / 0.680424 (-0.550221) | 0.016773 / 0.534201 (-0.517428) | 0.332909 / 0.579283 (-0.246374) | 0.347469 / 0.434364 (-0.086895) | 0.381364 / 0.540337 (-0.158973) | 0.541723 / 1.386936 (-0.845213) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005934 / 0.011353 (-0.005419) | 0.003573 / 0.011008 (-0.007435) | 0.062195 / 0.038508 (0.023687) | 0.059026 / 0.023109 (0.035917) | 0.413993 / 0.275898 (0.138095) | 0.459552 / 0.323480 (0.136072) | 0.004610 / 0.007986 (-0.003376) | 0.002907 / 0.004328 (-0.001421) | 0.062983 / 0.004250 (0.058733) | 0.047797 / 0.037052 (0.010745) | 0.415461 / 0.258489 (0.156972) | 0.417424 / 0.293841 (0.123583) | 0.027098 / 0.128546 (-0.101449) | 0.008106 / 0.075646 (-0.067540) | 0.067600 / 0.419271 (-0.351672) | 0.041432 / 0.043533 (-0.002101) | 0.407861 / 0.255139 (0.152722) | 0.430774 / 0.283200 (0.147575) | 0.020738 / 0.141683 (-0.120945) | 1.435127 / 1.452155 (-0.017028) | 1.486961 / 1.492716 (-0.005755) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.231174 / 0.018006 (0.213168) | 0.421208 / 0.000490 (0.420718) | 0.005411 / 0.000200 (0.005211) | 0.000078 / 0.000054 (0.000023) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025362 / 0.037411 (-0.012049) | 0.078534 / 0.014526 (0.064008) | 0.085304 / 0.176557 (-0.091252) | 0.139048 / 0.737135 (-0.598087) | 0.087015 / 0.296338 (-0.209323) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.448506 / 0.215209 (0.233297) | 4.486694 / 2.077655 (2.409039) | 2.488022 / 1.504120 (0.983902) | 2.325321 / 1.541195 (0.784126) | 2.381311 / 1.468490 (0.912821) | 0.502102 / 4.584777 (-4.082675) | 3.018326 / 3.745712 (-0.727386) | 2.824922 / 5.269862 (-2.444940) | 1.857414 / 4.565676 (-2.708263) | 0.057514 / 0.424275 (-0.366761) | 0.006829 / 0.007607 (-0.000779) | 0.521939 / 0.226044 (0.295895) | 5.224393 / 2.268929 (2.955465) | 2.933132 / 55.444624 (-52.511492) | 2.661187 / 6.876477 (-4.215290) | 2.781950 / 2.142072 (0.639878) | 0.592927 / 4.805227 (-4.212300) | 0.126685 / 6.500664 (-6.373979) | 0.064188 / 0.075469 (-0.011281) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.351107 / 1.841788 (-0.490681) | 18.344453 / 8.074308 (10.270145) | 13.838788 / 10.191392 (3.647396) | 0.157881 / 0.680424 (-0.522543) | 0.016636 / 0.534201 (-0.517565) | 0.331597 / 0.579283 (-0.247686) | 0.345573 / 0.434364 (-0.088791) | 0.397361 / 0.540337 (-0.142976) | 0.534289 / 1.386936 (-0.852647) |

</details>
</details>

![](https://cml.dev/watermark.png#582e722a76534904c0f3038d32ebb8db88ce9128 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006399 / 0.011353 (-0.004954) | 0.003872 / 0.011008 (-0.007136) | 0.083722 / 0.038508 (0.045214) | 0.068845 / 0.023109 (0.045736) | 0.329112 / 0.275898 (0.053214) | 0.343295 / 0.323480 (0.019815) | 0.005137 / 0.007986 (-0.002849) | 0.003303 / 0.004328 (-0.001026) | 0.064495 / 0.004250 (0.060245) | 0.051448 / 0.037052 (0.014395) | 0.322554 / 0.258489 (0.064065) | 0.361934 / 0.293841 (0.068093) | 0.030821 / 0.128546 (-0.097726) | 0.008482 / 0.075646 (-0.067164) | 0.288136 / 0.419271 (-0.131135) | 0.051935 / 0.043533 (0.008402) | 0.308283 / 0.255139 (0.053144) | 0.343421 / 0.283200 (0.060221) | 0.023639 / 0.141683 (-0.118044) | 1.485442 / 1.452155 (0.033288) | 1.533282 / 1.492716 (0.040565) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.218163 / 0.018006 (0.200157) | 0.464473 / 0.000490 (0.463983) | 0.003097 / 0.000200 (0.002897) | 0.000081 / 0.000054 (0.000026) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028650 / 0.037411 (-0.008761) | 0.083295 / 0.014526 (0.068769) | 0.096468 / 0.176557 (-0.080088) | 0.152086 / 0.737135 (-0.585050) | 0.102586 / 0.296338 (-0.193752) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.393038 / 0.215209 (0.177829) | 3.925514 / 2.077655 (1.847859) | 1.938419 / 1.504120 (0.434300) | 1.760265 / 1.541195 (0.219071) | 1.810024 / 1.468490 (0.341534) | 0.486232 / 4.584777 (-4.098545) | 3.618747 / 3.745712 (-0.126965) | 3.206950 / 5.269862 (-2.062912) | 1.999240 / 4.565676 (-2.566436) | 0.056986 / 0.424275 (-0.367289) | 0.007193 / 0.007607 (-0.000415) | 0.469313 / 0.226044 (0.243269) | 4.688670 / 2.268929 (2.419741) | 2.400332 / 55.444624 (-53.044292) | 2.074197 / 6.876477 (-4.802279) | 2.290823 / 2.142072 (0.148751) | 0.582339 / 4.805227 (-4.222888) | 0.134127 / 6.500664 (-6.366537) | 0.061061 / 0.075469 (-0.014408) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.272782 / 1.841788 (-0.569006) | 19.463375 / 8.074308 (11.389067) | 14.306819 / 10.191392 (4.115427) | 0.164608 / 0.680424 (-0.515816) | 0.018626 / 0.534201 (-0.515575) | 0.395225 / 0.579283 (-0.184058) | 0.408984 / 0.434364 (-0.025380) | 0.463364 / 0.540337 (-0.076974) | 0.630425 / 1.386936 (-0.756511) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006465 / 0.011353 (-0.004888) | 0.003975 / 0.011008 (-0.007033) | 0.063643 / 0.038508 (0.025134) | 0.075214 / 0.023109 (0.052105) | 0.361734 / 0.275898 (0.085836) | 0.396664 / 0.323480 (0.073184) | 0.005251 / 0.007986 (-0.002735) | 0.003249 / 0.004328 (-0.001080) | 0.063841 / 0.004250 (0.059591) | 0.054504 / 0.037052 (0.017451) | 0.374791 / 0.258489 (0.116302) | 0.399205 / 0.293841 (0.105364) | 0.031355 / 0.128546 (-0.097192) | 0.008483 / 0.075646 (-0.067163) | 0.070234 / 0.419271 (-0.349037) | 0.048336 / 0.043533 (0.004803) | 0.373484 / 0.255139 (0.118345) | 0.382174 / 0.283200 (0.098974) | 0.022560 / 0.141683 (-0.119123) | 1.449799 / 1.452155 (-0.002355) | 1.525255 / 1.492716 (0.032539) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.228350 / 0.018006 (0.210343) | 0.444344 / 0.000490 (0.443855) | 0.003699 / 0.000200 (0.003499) | 0.000079 / 0.000054 (0.000024) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030681 / 0.037411 (-0.006731) | 0.087340 / 0.014526 (0.072814) | 0.098636 / 0.176557 (-0.077920) | 0.151665 / 0.737135 (-0.585471) | 0.100840 / 0.296338 (-0.195498) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.417857 / 0.215209 (0.202648) | 4.168407 / 2.077655 (2.090752) | 2.201758 / 1.504120 (0.697638) | 1.997834 / 1.541195 (0.456639) | 2.127693 / 1.468490 (0.659202) | 0.486429 / 4.584777 (-4.098348) | 3.676335 / 3.745712 (-0.069378) | 3.226268 / 5.269862 (-2.043594) | 2.027255 / 4.565676 (-2.538422) | 0.056759 / 0.424275 (-0.367516) | 0.007628 / 0.007607 (0.000021) | 0.500482 / 0.226044 (0.274438) | 4.996236 / 2.268929 (2.727307) | 2.628884 / 55.444624 (-52.815740) | 2.347611 / 6.876477 (-4.528866) | 2.551328 / 2.142072 (0.409255) | 0.582449 / 4.805227 (-4.222778) | 0.132844 / 6.500664 (-6.367821) | 0.061791 / 0.075469 (-0.013678) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.373718 / 1.841788 (-0.468070) | 19.921217 / 8.074308 (11.846909) | 14.209642 / 10.191392 (4.018250) | 0.185334 / 0.680424 (-0.495090) | 0.018228 / 0.534201 (-0.515973) | 0.395549 / 0.579283 (-0.183734) | 0.404446 / 0.434364 (-0.029918) | 0.472456 / 0.540337 (-0.067882) | 0.622739 / 1.386936 (-0.764197) |

</details>
</details>

![](https://cml.dev/watermark.png#33f736eafa0f77de03aa6894ea4a6c923702e5d1 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006007 / 0.011353 (-0.005346) | 0.003588 / 0.011008 (-0.007420) | 0.080334 / 0.038508 (0.041826) | 0.058932 / 0.023109 (0.035823) | 0.404613 / 0.275898 (0.128715) | 0.438377 / 0.323480 (0.114897) | 0.003468 / 0.007986 (-0.004518) | 0.003702 / 0.004328 (-0.000627) | 0.062936 / 0.004250 (0.058686) | 0.047987 / 0.037052 (0.010934) | 0.411409 / 0.258489 (0.152920) | 0.450244 / 0.293841 (0.156403) | 0.027007 / 0.128546 (-0.101539) | 0.007932 / 0.075646 (-0.067714) | 0.261390 / 0.419271 (-0.157882) | 0.044992 / 0.043533 (0.001459) | 0.409730 / 0.255139 (0.154591) | 0.433331 / 0.283200 (0.150131) | 0.020446 / 0.141683 (-0.121237) | 1.425418 / 1.452155 (-0.026736) | 1.479242 / 1.492716 (-0.013475) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.187375 / 0.018006 (0.169368) | 0.428532 / 0.000490 (0.428043) | 0.003406 / 0.000200 (0.003206) | 0.000072 / 0.000054 (0.000018) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024390 / 0.037411 (-0.013022) | 0.072571 / 0.014526 (0.058045) | 0.083513 / 0.176557 (-0.093044) | 0.144395 / 0.737135 (-0.592741) | 0.084813 / 0.296338 (-0.211526) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.409176 / 0.215209 (0.193967) | 4.078082 / 2.077655 (2.000428) | 1.913596 / 1.504120 (0.409476) | 1.718470 / 1.541195 (0.177275) | 1.753106 / 1.468490 (0.284616) | 0.494167 / 4.584777 (-4.090610) | 3.029531 / 3.745712 (-0.716181) | 2.807331 / 5.269862 (-2.462531) | 1.839471 / 4.565676 (-2.726206) | 0.057169 / 0.424275 (-0.367106) | 0.006433 / 0.007607 (-0.001175) | 0.482666 / 0.226044 (0.256621) | 4.817601 / 2.268929 (2.548673) | 2.449967 / 55.444624 (-52.994658) | 2.113891 / 6.876477 (-4.762586) | 2.399293 / 2.142072 (0.257221) | 0.578903 / 4.805227 (-4.226324) | 0.124306 / 6.500664 (-6.376358) | 0.061572 / 0.075469 (-0.013897) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.254692 / 1.841788 (-0.587096) | 18.414049 / 8.074308 (10.339741) | 13.992059 / 10.191392 (3.800667) | 0.146671 / 0.680424 (-0.533753) | 0.016925 / 0.534201 (-0.517275) | 0.333124 / 0.579283 (-0.246159) | 0.348007 / 0.434364 (-0.086357) | 0.378519 / 0.540337 (-0.161819) | 0.532540 / 1.386936 (-0.854396) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006050 / 0.011353 (-0.005303) | 0.003614 / 0.011008 (-0.007394) | 0.061707 / 0.038508 (0.023199) | 0.062874 / 0.023109 (0.039765) | 0.364760 / 0.275898 (0.088862) | 0.398136 / 0.323480 (0.074656) | 0.005598 / 0.007986 (-0.002388) | 0.002836 / 0.004328 (-0.001493) | 0.061880 / 0.004250 (0.057630) | 0.048165 / 0.037052 (0.011113) | 0.372656 / 0.258489 (0.114167) | 0.403967 / 0.293841 (0.110126) | 0.027046 / 0.128546 (-0.101501) | 0.008091 / 0.075646 (-0.067555) | 0.066783 / 0.419271 (-0.352489) | 0.041186 / 0.043533 (-0.002347) | 0.376009 / 0.255139 (0.120870) | 0.391769 / 0.283200 (0.108569) | 0.021020 / 0.141683 (-0.120663) | 1.514593 / 1.452155 (0.062438) | 1.548506 / 1.492716 (0.055790) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.237610 / 0.018006 (0.219604) | 0.434274 / 0.000490 (0.433784) | 0.009720 / 0.000200 (0.009520) | 0.000098 / 0.000054 (0.000043) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025605 / 0.037411 (-0.011807) | 0.078971 / 0.014526 (0.064445) | 0.088154 / 0.176557 (-0.088403) | 0.139112 / 0.737135 (-0.598023) | 0.088890 / 0.296338 (-0.207449) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.420027 / 0.215209 (0.204818) | 4.189493 / 2.077655 (2.111838) | 2.143907 / 1.504120 (0.639787) | 1.967032 / 1.541195 (0.425837) | 2.011845 / 1.468490 (0.543355) | 0.496692 / 4.584777 (-4.088085) | 3.025456 / 3.745712 (-0.720256) | 2.828436 / 5.269862 (-2.441426) | 1.860673 / 4.565676 (-2.705003) | 0.057199 / 0.424275 (-0.367076) | 0.006770 / 0.007607 (-0.000838) | 0.491281 / 0.226044 (0.265236) | 4.918065 / 2.268929 (2.649136) | 2.593172 / 55.444624 (-52.851452) | 2.250750 / 6.876477 (-4.625727) | 2.406235 / 2.142072 (0.264162) | 0.588648 / 4.805227 (-4.216579) | 0.125635 / 6.500664 (-6.375029) | 0.061697 / 0.075469 (-0.013773) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.374065 / 1.841788 (-0.467722) | 18.439315 / 8.074308 (10.365007) | 14.031660 / 10.191392 (3.840268) | 0.153665 / 0.680424 (-0.526759) | 0.016980 / 0.534201 (-0.517221) | 0.331799 / 0.579283 (-0.247484) | 0.343201 / 0.434364 (-0.091163) | 0.392445 / 0.540337 (-0.147892) | 0.530387 / 1.386936 (-0.856549) |

</details>
</details>

![](https://cml.dev/watermark.png#33f736eafa0f77de03aa6894ea4a6c923702e5d1 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008189 / 0.011353 (-0.003164) | 0.004598 / 0.011008 (-0.006410) | 0.102199 / 0.038508 (0.063691) | 0.077961 / 0.023109 (0.054852) | 0.364936 / 0.275898 (0.089038) | 0.402606 / 0.323480 (0.079126) | 0.005522 / 0.007986 (-0.002464) | 0.004007 / 0.004328 (-0.000322) | 0.071560 / 0.004250 (0.067310) | 0.055818 / 0.037052 (0.018765) | 0.378394 / 0.258489 (0.119905) | 0.428990 / 0.293841 (0.135149) | 0.043142 / 0.128546 (-0.085404) | 0.013254 / 0.075646 (-0.062392) | 0.331102 / 0.419271 (-0.088170) | 0.061407 / 0.043533 (0.017875) | 0.387397 / 0.255139 (0.132258) | 0.416062 / 0.283200 (0.132862) | 0.036330 / 0.141683 (-0.105353) | 1.735352 / 1.452155 (0.283198) | 1.773329 / 1.492716 (0.280613) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.188587 / 0.018006 (0.170581) | 0.519506 / 0.000490 (0.519016) | 0.004702 / 0.000200 (0.004502) | 0.000097 / 0.000054 (0.000043) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027152 / 0.037411 (-0.010260) | 0.094296 / 0.014526 (0.079770) | 0.098155 / 0.176557 (-0.078402) | 0.162541 / 0.737135 (-0.574595) | 0.112092 / 0.296338 (-0.184246) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.537555 / 0.215209 (0.322346) | 5.486821 / 2.077655 (3.409166) | 2.377127 / 1.504120 (0.873008) | 2.073205 / 1.541195 (0.532011) | 2.075130 / 1.468490 (0.606640) | 0.783779 / 4.584777 (-3.800998) | 5.029524 / 3.745712 (1.283812) | 4.382724 / 5.269862 (-0.887138) | 2.836180 / 4.565676 (-1.729496) | 0.108840 / 0.424275 (-0.315435) | 0.008123 / 0.007607 (0.000516) | 0.673460 / 0.226044 (0.447416) | 6.674030 / 2.268929 (4.405102) | 3.208922 / 55.444624 (-52.235702) | 2.464908 / 6.876477 (-4.411568) | 2.661929 / 2.142072 (0.519856) | 0.962529 / 4.805227 (-3.842698) | 0.197974 / 6.500664 (-6.302690) | 0.066656 / 0.075469 (-0.008813) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.430373 / 1.841788 (-0.411415) | 21.180540 / 8.074308 (13.106232) | 19.027491 / 10.191392 (8.836099) | 0.217520 / 0.680424 (-0.462904) | 0.028038 / 0.534201 (-0.506163) | 0.435266 / 0.579283 (-0.144017) | 0.529510 / 0.434364 (0.095147) | 0.511011 / 0.540337 (-0.029327) | 0.728940 / 1.386936 (-0.657996) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007883 / 0.011353 (-0.003470) | 0.004448 / 0.011008 (-0.006560) | 0.071350 / 0.038508 (0.032842) | 0.075269 / 0.023109 (0.052160) | 0.396705 / 0.275898 (0.120807) | 0.457809 / 0.323480 (0.134329) | 0.005193 / 0.007986 (-0.002792) | 0.003695 / 0.004328 (-0.000633) | 0.078087 / 0.004250 (0.073836) | 0.054276 / 0.037052 (0.017224) | 0.412184 / 0.258489 (0.153695) | 0.452400 / 0.293841 (0.158559) | 0.049762 / 0.128546 (-0.078784) | 0.013206 / 0.075646 (-0.062440) | 0.085985 / 0.419271 (-0.333287) | 0.058837 / 0.043533 (0.015304) | 0.432481 / 0.255139 (0.177342) | 0.433260 / 0.283200 (0.150060) | 0.031190 / 0.141683 (-0.110493) | 1.582707 / 1.452155 (0.130552) | 1.664457 / 1.492716 (0.171741) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.223639 / 0.018006 (0.205633) | 0.524388 / 0.000490 (0.523899) | 0.005489 / 0.000200 (0.005289) | 0.000099 / 0.000054 (0.000045) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030182 / 0.037411 (-0.007230) | 0.089309 / 0.014526 (0.074783) | 0.103306 / 0.176557 (-0.073250) | 0.162624 / 0.737135 (-0.574511) | 0.108957 / 0.296338 (-0.187381) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.577423 / 0.215209 (0.362214) | 5.900154 / 2.077655 (3.822500) | 2.687369 / 1.504120 (1.183249) | 2.513061 / 1.541195 (0.971866) | 2.506453 / 1.468490 (1.037963) | 0.830838 / 4.584777 (-3.753939) | 5.032195 / 3.745712 (1.286483) | 4.396827 / 5.269862 (-0.873035) | 2.884230 / 4.565676 (-1.681447) | 0.102239 / 0.424275 (-0.322036) | 0.008178 / 0.007607 (0.000571) | 0.710027 / 0.226044 (0.483983) | 7.149626 / 2.268929 (4.880698) | 3.403605 / 55.444624 (-52.041019) | 2.661970 / 6.876477 (-4.214506) | 2.760227 / 2.142072 (0.618154) | 1.043981 / 4.805227 (-3.761246) | 0.195028 / 6.500664 (-6.305636) | 0.065211 / 0.075469 (-0.010258) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.581265 / 1.841788 (-0.260522) | 21.640230 / 8.074308 (13.565922) | 19.031860 / 10.191392 (8.840468) | 0.196903 / 0.680424 (-0.483520) | 0.027061 / 0.534201 (-0.507140) | 0.444995 / 0.579283 (-0.134288) | 0.528195 / 0.434364 (0.093831) | 0.521540 / 0.540337 (-0.018797) | 0.730204 / 1.386936 (-0.656732) |

</details>
</details>

![](https://cml.dev/watermark.png#33f736eafa0f77de03aa6894ea4a6c923702e5d1 "CML watermark")