Release: 2.9.0
Opened by lhoestq (closed)


Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008578 / 0.011353 (-0.002775) | 0.004535 / 0.011008 (-0.006473) | 0.100694 / 0.038508 (0.062186) | 0.029570 / 0.023109 (0.006460) | 0.296384 / 0.275898 (0.020486) | 0.354405 / 0.323480 (0.030925) | 0.006962 / 0.007986 (-0.001024) | 0.003405 / 0.004328 (-0.000924) | 0.077275 / 0.004250 (0.073025) | 0.036623 / 0.037052 (-0.000429) | 0.309844 / 0.258489 (0.051355) | 0.340343 / 0.293841 (0.046502) | 0.033626 / 0.128546 (-0.094920) | 0.011433 / 0.075646 (-0.064214) | 0.322659 / 0.419271 (-0.096612) | 0.040509 / 0.043533 (-0.003024) | 0.294002 / 0.255139 (0.038863) | 0.323259 / 0.283200 (0.040059) | 0.088023 / 0.141683 (-0.053660) | 1.462039 / 1.452155 (0.009885) | 1.495401 / 1.492716 (0.002684) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.218614 / 0.018006 (0.200608) | 0.482359 / 0.000490 (0.481869) | 0.001216 / 0.000200 (0.001016) | 0.000081 / 0.000054 (0.000027) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023167 / 0.037411 (-0.014245) | 0.098468 / 0.014526 (0.083942) | 0.108273 / 0.176557 (-0.068284) | 0.139991 / 0.737135 (-0.597144) | 0.109032 / 0.296338 (-0.187307) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.421526 / 0.215209 (0.206317) | 4.216808 / 2.077655 (2.139153) | 1.860550 / 1.504120 (0.356431) | 1.654518 / 1.541195 (0.113323) | 1.699064 / 1.468490 (0.230574) | 0.691489 / 4.584777 (-3.893287) | 3.401885 / 3.745712 (-0.343827) | 2.792860 / 5.269862 (-2.477001) | 1.516269 / 4.565676 (-3.049408) | 0.081627 / 0.424275 (-0.342648) | 0.012556 / 0.007607 (0.004949) | 0.531535 / 0.226044 (0.305491) | 5.320752 / 2.268929 (3.051823) | 2.314502 / 55.444624 (-53.130123) | 1.967118 / 6.876477 (-4.909359) | 2.008252 / 2.142072 (-0.133821) | 0.809730 / 4.805227 (-3.995497) | 0.148112 / 6.500664 (-6.352552) | 0.064821 / 0.075469 (-0.010648) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.269754 / 1.841788 (-0.572033) | 13.884200 / 8.074308 (5.809892) | 13.914390 / 10.191392 (3.722998) | 0.150176 / 0.680424 (-0.530248) | 0.028463 / 0.534201 (-0.505738) | 0.398723 / 0.579283 (-0.180561) | 0.400433 / 0.434364 (-0.033931) | 0.485169 / 0.540337 (-0.055169) | 0.565995 / 1.386936 (-0.820941) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006479 / 0.011353 (-0.004874) | 0.004504 / 0.011008 (-0.006504) | 0.097905 / 0.038508 (0.059397) | 0.027140 / 0.023109 (0.004031) | 0.408742 / 0.275898 (0.132844) | 0.448707 / 0.323480 (0.125228) | 0.004819 / 0.007986 (-0.003166) | 0.004761 / 0.004328 (0.000433) | 0.075456 / 0.004250 (0.071205) | 0.036282 / 0.037052 (-0.000771) | 0.405961 / 0.258489 (0.147472) | 0.449411 / 0.293841 (0.155570) | 0.031159 / 0.128546 (-0.097387) | 0.011693 / 0.075646 (-0.063954) | 0.321124 / 0.419271 (-0.098147) | 0.041369 / 0.043533 (-0.002164) | 0.408070 / 0.255139 (0.152931) | 0.428704 / 0.283200 (0.145504) | 0.086839 / 0.141683 (-0.054844) | 1.477772 / 1.452155 (0.025617) | 1.555913 / 1.492716 (0.063197) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.239494 / 0.018006 (0.221488) | 0.410785 / 0.000490 (0.410295) | 0.000989 / 0.000200 (0.000789) | 0.000072 / 0.000054 (0.000017) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023805 / 0.037411 (-0.013607) | 0.097904 / 0.014526 (0.083378) | 0.106437 / 0.176557 (-0.070120) | 0.140555 / 0.737135 (-0.596580) | 0.107169 / 0.296338 (-0.189170) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.470233 / 0.215209 (0.255024) | 4.700451 / 2.077655 (2.622797) | 2.391712 / 1.504120 (0.887592) | 2.191125 / 1.541195 (0.649930) | 2.268924 / 1.468490 (0.800434) | 0.692421 / 4.584777 (-3.892356) | 3.387117 / 3.745712 (-0.358595) | 1.881731 / 5.269862 (-3.388130) | 1.155759 / 4.565676 (-3.409917) | 0.082040 / 0.424275 (-0.342236) | 0.012687 / 0.007607 (0.005080) | 0.567556 / 0.226044 (0.341511) | 5.701408 / 2.268929 (3.432480) | 2.864368 / 55.444624 (-52.580256) | 2.512073 / 6.876477 (-4.364404) | 2.546078 / 2.142072 (0.404005) | 0.795939 / 4.805227 (-4.009288) | 0.150078 / 6.500664 (-6.350586) | 0.067644 / 0.075469 (-0.007825) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.281681 / 1.841788 (-0.560107) | 13.967107 / 8.074308 (5.892799) | 13.293648 / 10.191392 (3.102256) | 0.128027 / 0.680424 (-0.552397) | 0.016791 / 0.534201 (-0.517410) | 0.379400 / 0.579283 (-0.199884) | 0.386847 / 0.434364 (-0.047517) | 0.469859 / 0.540337 (-0.070478) | 0.564203 / 1.386936 (-0.822733) |

</details>
</details>

![](https://cml.dev/watermark.png#90832b5e33774ea8ec35ccb92ac14649a345bdbe "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008701 / 0.011353 (-0.002652) | 0.004564 / 0.011008 (-0.006444) | 0.100578 / 0.038508 (0.062070) | 0.029209 / 0.023109 (0.006100) | 0.315308 / 0.275898 (0.039410) | 0.381022 / 0.323480 (0.057542) | 0.007152 / 0.007986 (-0.000834) | 0.003511 / 0.004328 (-0.000817) | 0.078361 / 0.004250 (0.074110) | 0.035394 / 0.037052 (-0.001658) | 0.331076 / 0.258489 (0.072586) | 0.366613 / 0.293841 (0.072772) | 0.033466 / 0.128546 (-0.095080) | 0.011521 / 0.075646 (-0.064126) | 0.322178 / 0.419271 (-0.097093) | 0.040891 / 0.043533 (-0.002641) | 0.320418 / 0.255139 (0.065279) | 0.345199 / 0.283200 (0.062000) | 0.087906 / 0.141683 (-0.053777) | 1.476801 / 1.452155 (0.024646) | 1.497738 / 1.492716 (0.005022) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.178094 / 0.018006 (0.160087) | 0.408317 / 0.000490 (0.407827) | 0.001825 / 0.000200 (0.001625) | 0.000067 / 0.000054 (0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022402 / 0.037411 (-0.015010) | 0.097104 / 0.014526 (0.082578) | 0.105361 / 0.176557 (-0.071196) | 0.139728 / 0.737135 (-0.597407) | 0.109613 / 0.296338 (-0.186725) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.418245 / 0.215209 (0.203036) | 4.155655 / 2.077655 (2.078000) | 1.865892 / 1.504120 (0.361772) | 1.659003 / 1.541195 (0.117809) | 1.725649 / 1.468490 (0.257159) | 0.688733 / 4.584777 (-3.896044) | 3.323529 / 3.745712 (-0.422184) | 1.867807 / 5.269862 (-3.402054) | 1.157740 / 4.565676 (-3.407936) | 0.081947 / 0.424275 (-0.342329) | 0.012471 / 0.007607 (0.004864) | 0.529333 / 0.226044 (0.303288) | 5.284898 / 2.268929 (3.015970) | 2.321741 / 55.444624 (-53.122883) | 1.975683 / 6.876477 (-4.900794) | 2.029691 / 2.142072 (-0.112381) | 0.810212 / 4.805227 (-3.995015) | 0.148185 / 6.500664 (-6.352479) | 0.064594 / 0.075469 (-0.010875) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.183391 / 1.841788 (-0.658396) | 13.574760 / 8.074308 (5.500452) | 14.215015 / 10.191392 (4.023623) | 0.150776 / 0.680424 (-0.529648) | 0.029058 / 0.534201 (-0.505143) | 0.404071 / 0.579283 (-0.175212) | 0.401289 / 0.434364 (-0.033075) | 0.490946 / 0.540337 (-0.049392) | 0.582292 / 1.386936 (-0.804644) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006695 / 0.011353 (-0.004658) | 0.004499 / 0.011008 (-0.006510) | 0.097633 / 0.038508 (0.059125) | 0.027606 / 0.023109 (0.004496) | 0.413191 / 0.275898 (0.137293) | 0.441896 / 0.323480 (0.118416) | 0.005703 / 0.007986 (-0.002283) | 0.004608 / 0.004328 (0.000280) | 0.074392 / 0.004250 (0.070141) | 0.037966 / 0.037052 (0.000913) | 0.410736 / 0.258489 (0.152247) | 0.448581 / 0.293841 (0.154740) | 0.031594 / 0.128546 (-0.096952) | 0.011597 / 0.075646 (-0.064049) | 0.319632 / 0.419271 (-0.099639) | 0.041189 / 0.043533 (-0.002343) | 0.407120 / 0.255139 (0.151981) | 0.433416 / 0.283200 (0.150216) | 0.089932 / 0.141683 (-0.051751) | 1.453919 / 1.452155 (0.001764) | 1.545892 / 1.492716 (0.053176) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.224302 / 0.018006 (0.206296) | 0.415519 / 0.000490 (0.415029) | 0.000407 / 0.000200 (0.000207) | 0.000060 / 0.000054 (0.000006) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024104 / 0.037411 (-0.013307) | 0.098202 / 0.014526 (0.083676) | 0.106416 / 0.176557 (-0.070140) | 0.141090 / 0.737135 (-0.596045) | 0.110188 / 0.296338 (-0.186150) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.478252 / 0.215209 (0.263043) | 4.739684 / 2.077655 (2.662029) | 2.419040 / 1.504120 (0.914920) | 2.217705 / 1.541195 (0.676510) | 2.303288 / 1.468490 (0.834798) | 0.696682 / 4.584777 (-3.888095) | 3.401962 / 3.745712 (-0.343750) | 1.886015 / 5.269862 (-3.383846) | 1.175084 / 4.565676 (-3.390592) | 0.083064 / 0.424275 (-0.341211) | 0.012613 / 0.007607 (0.005006) | 0.579105 / 0.226044 (0.353060) | 5.792119 / 2.268929 (3.523191) | 2.889778 / 55.444624 (-52.554846) | 2.537438 / 6.876477 (-4.339039) | 2.574814 / 2.142072 (0.432741) | 0.803438 / 4.805227 (-4.001789) | 0.151912 / 6.500664 (-6.348752) | 0.068291 / 0.075469 (-0.007178) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.286002 / 1.841788 (-0.555786) | 14.179443 / 8.074308 (6.105135) | 13.443939 / 10.191392 (3.252547) | 0.152427 / 0.680424 (-0.527996) | 0.017248 / 0.534201 (-0.516953) | 0.378734 / 0.579283 (-0.200549) | 0.382276 / 0.434364 (-0.052087) | 0.465323 / 0.540337 (-0.075014) | 0.556454 / 1.386936 (-0.830482) |

</details>
</details>

![](https://cml.dev/watermark.png#b5672a956d5de864e6f5550e493527d962d6ae55 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008675 / 0.011353 (-0.002678) | 0.004537 / 0.011008 (-0.006471) | 0.100179 / 0.038508 (0.061671) | 0.029307 / 0.023109 (0.006198) | 0.294687 / 0.275898 (0.018789) | 0.356868 / 0.323480 (0.033388) | 0.006992 / 0.007986 (-0.000994) | 0.003380 / 0.004328 (-0.000949) | 0.076961 / 0.004250 (0.072710) | 0.036047 / 0.037052 (-0.001005) | 0.308037 / 0.258489 (0.049548) | 0.341089 / 0.293841 (0.047248) | 0.033416 / 0.128546 (-0.095131) | 0.011534 / 0.075646 (-0.064112) | 0.322976 / 0.419271 (-0.096296) | 0.040894 / 0.043533 (-0.002639) | 0.296501 / 0.255139 (0.041362) | 0.324605 / 0.283200 (0.041405) | 0.086713 / 0.141683 (-0.054970) | 1.502784 / 1.452155 (0.050630) | 1.535013 / 1.492716 (0.042297) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.186647 / 0.018006 (0.168641) | 0.411003 / 0.000490 (0.410514) | 0.003594 / 0.000200 (0.003394) | 0.000074 / 0.000054 (0.000020) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023704 / 0.037411 (-0.013707) | 0.096154 / 0.014526 (0.081629) | 0.103671 / 0.176557 (-0.072885) | 0.138878 / 0.737135 (-0.598258) | 0.106947 / 0.296338 (-0.189391) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.417180 / 0.215209 (0.201970) | 4.149579 / 2.077655 (2.071925) | 1.865763 / 1.504120 (0.361643) | 1.669722 / 1.541195 (0.128527) | 1.722345 / 1.468490 (0.253855) | 0.695910 / 4.584777 (-3.888867) | 3.342266 / 3.745712 (-0.403446) | 1.884568 / 5.269862 (-3.385294) | 1.265013 / 4.565676 (-3.300664) | 0.081836 / 0.424275 (-0.342439) | 0.012371 / 0.007607 (0.004764) | 0.522997 / 0.226044 (0.296953) | 5.225434 / 2.268929 (2.956506) | 2.304701 / 55.444624 (-53.139924) | 1.949067 / 6.876477 (-4.927410) | 2.016347 / 2.142072 (-0.125725) | 0.809850 / 4.805227 (-3.995377) | 0.148396 / 6.500664 (-6.352268) | 0.063340 / 0.075469 (-0.012129) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.224621 / 1.841788 (-0.617167) | 13.814223 / 8.074308 (5.739915) | 13.879728 / 10.191392 (3.688336) | 0.149530 / 0.680424 (-0.530894) | 0.028439 / 0.534201 (-0.505762) | 0.392726 / 0.579283 (-0.186557) | 0.396894 / 0.434364 (-0.037469) | 0.474395 / 0.540337 (-0.065943) | 0.569090 / 1.386936 (-0.817847) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006483 / 0.011353 (-0.004870) | 0.004527 / 0.011008 (-0.006481) | 0.098038 / 0.038508 (0.059530) | 0.027239 / 0.023109 (0.004130) | 0.441773 / 0.275898 (0.165875) | 0.471448 / 0.323480 (0.147968) | 0.005034 / 0.007986 (-0.002951) | 0.004732 / 0.004328 (0.000403) | 0.075036 / 0.004250 (0.070785) | 0.036711 / 0.037052 (-0.000341) | 0.442634 / 0.258489 (0.184145) | 0.476479 / 0.293841 (0.182638) | 0.031303 / 0.128546 (-0.097243) | 0.011642 / 0.075646 (-0.064005) | 0.320750 / 0.419271 (-0.098521) | 0.048698 / 0.043533 (0.005165) | 0.441205 / 0.255139 (0.186066) | 0.464845 / 0.283200 (0.181645) | 0.092716 / 0.141683 (-0.048967) | 1.510028 / 1.452155 (0.057874) | 1.574065 / 1.492716 (0.081349) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.220756 / 0.018006 (0.202750) | 0.393971 / 0.000490 (0.393482) | 0.002506 / 0.000200 (0.002306) | 0.000073 / 0.000054 (0.000018) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024455 / 0.037411 (-0.012956) | 0.100164 / 0.014526 (0.085638) | 0.108053 / 0.176557 (-0.068504) | 0.142973 / 0.737135 (-0.594163) | 0.110108 / 0.296338 (-0.186231) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.473639 / 0.215209 (0.258430) | 4.737521 / 2.077655 (2.659866) | 2.466208 / 1.504120 (0.962088) | 2.272608 / 1.541195 (0.731413) | 2.349255 / 1.468490 (0.880764) | 0.699928 / 4.584777 (-3.884849) | 3.348443 / 3.745712 (-0.397269) | 2.604611 / 5.269862 (-2.665250) | 1.543080 / 4.565676 (-3.022597) | 0.082627 / 0.424275 (-0.341648) | 0.012251 / 0.007607 (0.004644) | 0.569949 / 0.226044 (0.343905) | 5.732316 / 2.268929 (3.463388) | 2.913541 / 55.444624 (-52.531084) | 2.560584 / 6.876477 (-4.315892) | 2.615192 / 2.142072 (0.473120) | 0.803822 / 4.805227 (-4.001406) | 0.150821 / 6.500664 (-6.349843) | 0.067128 / 0.075469 (-0.008341) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.272278 / 1.841788 (-0.569510) | 13.783339 / 8.074308 (5.709030) | 13.243601 / 10.191392 (3.052209) | 0.136421 / 0.680424 (-0.544003) | 0.016565 / 0.534201 (-0.517636) | 0.381102 / 0.579283 (-0.198181) | 0.386166 / 0.434364 (-0.048197) | 0.474249 / 0.540337 (-0.066089) | 0.566826 / 1.386936 (-0.820110) |

</details>
</details>

![](https://cml.dev/watermark.png#b5672a956d5de864e6f5550e493527d962d6ae55 "CML watermark")