Release: 2.14.7
Opened by albertvillanova (closed)

Release 2.14.7.

Comments:
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004943 / 0.011353 (-0.006410) | 0.002900 / 0.011008 (-0.008109) | 0.061495 / 0.038508 (0.022987) | 0.053575 / 0.023109 (0.030466) | 0.249318 / 0.275898 (-0.026580) | 0.271773 / 0.323480 (-0.051706) | 0.003074 / 0.007986 (-0.004911) | 0.003738 / 0.004328 (-0.000590) | 0.047624 / 0.004250 (0.043373) | 0.045141 / 0.037052 (0.008089) | 0.255467 / 0.258489 (-0.003022) | 0.286577 / 0.293841 (-0.007264) | 0.023113 / 0.128546 (-0.105433) | 0.007189 / 0.075646 (-0.068458) | 0.204441 / 0.419271 (-0.214830) | 0.036829 / 0.043533 (-0.006704) | 0.252474 / 0.255139 (-0.002665) | 0.270960 / 0.283200 (-0.012239) | 0.019666 / 0.141683 (-0.122017) | 1.095139 / 1.452155 (-0.357015) | 1.158659 / 1.492716 (-0.334057) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091046 / 0.018006 (0.073040) | 0.298346 / 0.000490 (0.297856) | 0.000215 / 0.000200 (0.000015) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018702 / 0.037411 (-0.018709) | 0.062213 / 0.014526 (0.047687) | 0.073364 / 0.176557 (-0.103193) | 0.119841 / 0.737135 (-0.617294) | 0.074070 / 0.296338 (-0.222268) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.282388 / 0.215209 (0.067179) | 2.792029 / 2.077655 (0.714375) | 1.471483 / 1.504120 (-0.032637) | 1.386236 / 1.541195 (-0.154959) | 1.377489 / 1.468490 (-0.091001) | 0.410335 / 4.584777 (-4.174442) | 2.424866 / 3.745712 (-1.320846) | 2.610609 / 5.269862 (-2.659253) | 1.574636 / 4.565676 (-2.991041) | 0.046716 / 0.424275 (-0.377559) | 0.004768 / 0.007607 (-0.002839) | 0.339831 / 0.226044 (0.113787) | 3.297579 / 2.268929 (1.028651) | 1.851410 / 55.444624 (-53.593214) | 1.550048 / 6.876477 (-5.326428) | 1.576647 / 2.142072 (-0.565425) | 0.482538 / 4.805227 (-4.322689) | 0.101381 / 6.500664 (-6.399283) | 0.042066 / 0.075469 (-0.033403) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.972664 / 1.841788 (-0.869123) | 11.580700 / 8.074308 (3.506392) | 10.586747 / 10.191392 (0.395355) | 0.127844 / 0.680424 (-0.552580) | 0.014270 / 0.534201 (-0.519931) | 0.269678 / 0.579283 (-0.309605) | 0.264022 / 0.434364 (-0.170342) | 0.309395 / 0.540337 (-0.230942) | 0.429228 / 1.386936 (-0.957708) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004815 / 0.011353 (-0.006538) | 0.002890 / 0.011008 (-0.008119) | 0.048039 / 0.038508 (0.009531) | 0.053029 / 0.023109 (0.029920) | 0.271346 / 0.275898 (-0.004552) | 0.294488 / 0.323480 (-0.028992) | 0.003983 / 0.007986 (-0.004003) | 0.002439 / 0.004328 (-0.001889) | 0.048250 / 0.004250 (0.044000) | 0.038855 / 0.037052 (0.001803) | 0.284723 / 0.258489 (0.026234) | 0.303604 / 0.293841 (0.009763) | 0.024384 / 0.128546 (-0.104163) | 0.007021 / 0.075646 (-0.068625) | 0.053850 / 0.419271 (-0.365422) | 0.032177 / 0.043533 (-0.011356) | 0.270039 / 0.255139 (0.014900) | 0.289669 / 0.283200 (0.006469) | 0.018840 / 0.141683 (-0.122842) | 1.122191 / 1.452155 (-0.329963) | 1.187083 / 1.492716 (-0.305634) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090609 / 0.018006 (0.072603) | 0.298915 / 0.000490 (0.298425) | 0.000216 / 0.000200 (0.000016) | 0.000043 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.020919 / 0.037411 (-0.016492) | 0.070474 / 0.014526 (0.055948) | 0.082421 / 0.176557 (-0.094135) | 0.126967 / 0.737135 (-0.610168) | 0.083447 / 0.296338 (-0.212892) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.300153 / 0.215209 (0.084944) | 2.958992 / 2.077655 (0.881337) | 1.631228 / 1.504120 (0.127108) | 1.497991 / 1.541195 (-0.043204) | 1.536963 / 1.468490 (0.068473) | 0.403047 / 4.584777 (-4.181730) | 2.448782 / 3.745712 (-1.296930) | 2.571954 / 5.269862 (-2.697908) | 1.556346 / 4.565676 (-3.009331) | 0.045992 / 0.424275 (-0.378283) | 0.004785 / 0.007607 (-0.002822) | 0.357448 / 0.226044 (0.131404) | 3.558808 / 2.268929 (1.289880) | 1.992624 / 55.444624 (-53.452001) | 1.695027 / 6.876477 (-5.181450) | 1.695183 / 2.142072 (-0.446889) | 0.477001 / 4.805227 (-4.328226) | 0.097485 / 6.500664 (-6.403179) | 0.040530 / 0.075469 (-0.034939) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.976342 / 1.841788 (-0.865445) | 12.141698 / 8.074308 (4.067390) | 10.881101 / 10.191392 (0.689709) | 0.142443 / 0.680424 (-0.537981) | 0.015583 / 0.534201 (-0.518618) | 0.269727 / 0.579283 (-0.309556) | 0.275890 / 0.434364 (-0.158474) | 0.306351 / 0.540337 (-0.233987) | 0.412003 / 1.386936 (-0.974933) |

</details>
</details>

![](https://cml.dev/watermark.png#7c744261000fd684f54c54de8ac4f15a726092d7 "CML watermark")


_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004946 / 0.011353 (-0.006407) | 0.002863 / 0.011008 (-0.008146) | 0.061888 / 0.038508 (0.023380) | 0.050664 / 0.023109 (0.027554) | 0.242635 / 0.275898 (-0.033263) | 0.271741 / 0.323480 (-0.051739) | 0.003023 / 0.007986 (-0.004963) | 0.003088 / 0.004328 (-0.001241) | 0.049286 / 0.004250 (0.045036) | 0.044699 / 0.037052 (0.007647) | 0.249581 / 0.258489 (-0.008908) | 0.285633 / 0.293841 (-0.008208) | 0.023048 / 0.128546 (-0.105499) | 0.007235 / 0.075646 (-0.068412) | 0.202989 / 0.419271 (-0.216282) | 0.036357 / 0.043533 (-0.007175) | 0.245980 / 0.255139 (-0.009159) | 0.277486 / 0.283200 (-0.005713) | 0.019215 / 0.141683 (-0.122468) | 1.096456 / 1.452155 (-0.355699) | 1.152196 / 1.492716 (-0.340520) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092026 / 0.018006 (0.074020) | 0.303038 / 0.000490 (0.302549) | 0.000209 / 0.000200 (0.000009) | 0.000048 / 0.000054 (-0.000006) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018670 / 0.037411 (-0.018741) | 0.061972 / 0.014526 (0.047446) | 0.072963 / 0.176557 (-0.103594) | 0.119984 / 0.737135 (-0.617151) | 0.074074 / 0.296338 (-0.222265) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.282444 / 0.215209 (0.067235) | 2.754571 / 2.077655 (0.676916) | 1.482635 / 1.504120 (-0.021485) | 1.352039 / 1.541195 (-0.189155) | 1.359333 / 1.468490 (-0.109157) | 0.399690 / 4.584777 (-4.185087) | 2.364844 / 3.745712 (-1.380868) | 2.603942 / 5.269862 (-2.665919) | 1.569512 / 4.565676 (-2.996164) | 0.046074 / 0.424275 (-0.378201) | 0.004745 / 0.007607 (-0.002862) | 0.339066 / 0.226044 (0.113022) | 3.363456 / 2.268929 (1.094527) | 1.822213 / 55.444624 (-53.622411) | 1.536622 / 6.876477 (-5.339854) | 1.574772 / 2.142072 (-0.567300) | 0.474418 / 4.805227 (-4.330809) | 0.099572 / 6.500664 (-6.401092) | 0.041824 / 0.075469 (-0.033645) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.956300 / 1.841788 (-0.885487) | 11.648886 / 8.074308 (3.574578) | 10.645700 / 10.191392 (0.454308) | 0.138924 / 0.680424 (-0.541499) | 0.013936 / 0.534201 (-0.520265) | 0.270319 / 0.579283 (-0.308964) | 0.269735 / 0.434364 (-0.164629) | 0.309699 / 0.540337 (-0.230639) | 0.429139 / 1.386936 (-0.957797) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004838 / 0.011353 (-0.006515) | 0.002937 / 0.011008 (-0.008072) | 0.048094 / 0.038508 (0.009586) | 0.053131 / 0.023109 (0.030022) | 0.271893 / 0.275898 (-0.004005) | 0.291025 / 0.323480 (-0.032454) | 0.004058 / 0.007986 (-0.003928) | 0.002410 / 0.004328 (-0.001919) | 0.047939 / 0.004250 (0.043689) | 0.038996 / 0.037052 (0.001944) | 0.274983 / 0.258489 (0.016494) | 0.306175 / 0.293841 (0.012334) | 0.024388 / 0.128546 (-0.104159) | 0.007242 / 0.075646 (-0.068404) | 0.054011 / 0.419271 (-0.365261) | 0.032750 / 0.043533 (-0.010783) | 0.271147 / 0.255139 (0.016008) | 0.288163 / 0.283200 (0.004963) | 0.018383 / 0.141683 (-0.123299) | 1.116134 / 1.452155 (-0.336021) | 1.185964 / 1.492716 (-0.306752) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093289 / 0.018006 (0.075283) | 0.303058 / 0.000490 (0.302568) | 0.000241 / 0.000200 (0.000041) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021422 / 0.037411 (-0.015990) | 0.069974 / 0.014526 (0.055449) | 0.081164 / 0.176557 (-0.095392) | 0.119991 / 0.737135 (-0.617144) | 0.082154 / 0.296338 (-0.214184) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.292298 / 0.215209 (0.077089) | 2.851475 / 2.077655 (0.773821) | 1.558283 / 1.504120 (0.054163) | 1.432431 / 1.541195 (-0.108764) | 1.479282 / 1.468490 (0.010792) | 0.413124 / 4.584777 (-4.171653) | 2.473005 / 3.745712 (-1.272707) | 2.548779 / 5.269862 (-2.721082) | 1.520776 / 4.565676 (-3.044900) | 0.046476 / 0.424275 (-0.377799) | 0.004814 / 0.007607 (-0.002794) | 0.347036 / 0.226044 (0.120992) | 3.424928 / 2.268929 (1.155999) | 1.963274 / 55.444624 (-53.481351) | 1.653794 / 6.876477 (-5.222683) | 1.643874 / 2.142072 (-0.498198) | 0.469086 / 4.805227 (-4.336141) | 0.097417 / 6.500664 (-6.403247) | 0.040468 / 0.075469 (-0.035002) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.972783 / 1.841788 (-0.869005) | 12.122994 / 8.074308 (4.048686) | 10.876396 / 10.191392 (0.685004) | 0.130573 / 0.680424 (-0.549850) | 0.016693 / 0.534201 (-0.517508) | 0.270952 / 0.579283 (-0.308331) | 0.273834 / 0.434364 (-0.160530) | 0.305049 / 0.540337 (-0.235289) | 0.408776 / 1.386936 (-0.978160) |

</details>
</details>

![](https://cml.dev/watermark.png#e4216e5d57ea07e6b1ed73a3ec2cf845c6e59f70 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004606 / 0.011353 (-0.006747) | 0.002433 / 0.011008 (-0.008576) | 0.061985 / 0.038508 (0.023477) | 0.048853 / 0.023109 (0.025744) | 0.244506 / 0.275898 (-0.031392) | 0.270159 / 0.323480 (-0.053321) | 0.003962 / 0.007986 (-0.004024) | 0.002376 / 0.004328 (-0.001952) | 0.048067 / 0.004250 (0.043817) | 0.041864 / 0.037052 (0.004812) | 0.249743 / 0.258489 (-0.008746) | 0.287723 / 0.293841 (-0.006117) | 0.022954 / 0.128546 (-0.105593) | 0.006845 / 0.075646 (-0.068801) | 0.206313 / 0.419271 (-0.212959) | 0.035780 / 0.043533 (-0.007753) | 0.244286 / 0.255139 (-0.010853) | 0.270026 / 0.283200 (-0.013173) | 0.018177 / 0.141683 (-0.123506) | 1.083998 / 1.452155 (-0.368157) | 1.156086 / 1.492716 (-0.336630) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093754 / 0.018006 (0.075748) | 0.302157 / 0.000490 (0.301667) | 0.000215 / 0.000200 (0.000015) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018745 / 0.037411 (-0.018666) | 0.061707 / 0.014526 (0.047181) | 0.074356 / 0.176557 (-0.102200) | 0.121643 / 0.737135 (-0.615492) | 0.075885 / 0.296338 (-0.220454) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.289156 / 0.215209 (0.073947) | 2.881327 / 2.077655 (0.803672) | 1.483568 / 1.504120 (-0.020552) | 1.355933 / 1.541195 (-0.185262) | 1.389693 / 1.468490 (-0.078797) | 0.402834 / 4.584777 (-4.181943) | 2.390634 / 3.745712 (-1.355078) | 2.596761 / 5.269862 (-2.673101) | 1.527602 / 4.565676 (-3.038074) | 0.046434 / 0.424275 (-0.377841) | 0.004783 / 0.007607 (-0.002824) | 0.341017 / 0.226044 (0.114972) | 3.429023 / 2.268929 (1.160095) | 1.832988 / 55.444624 (-53.611637) | 1.526510 / 6.876477 (-5.349967) | 1.539382 / 2.142072 (-0.602690) | 0.475734 / 4.805227 (-4.329493) | 0.098710 / 6.500664 (-6.401954) | 0.041136 / 0.075469 (-0.034333) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.922023 / 1.841788 (-0.919765) | 11.428215 / 8.074308 (3.353907) | 10.356668 / 10.191392 (0.165276) | 0.139575 / 0.680424 (-0.540848) | 0.014541 / 0.534201 (-0.519660) | 0.271359 / 0.579283 (-0.307924) | 0.266701 / 0.434364 (-0.167663) | 0.309449 / 0.540337 (-0.230888) | 0.422047 / 1.386936 (-0.964889) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004892 / 0.011353 (-0.006461) | 0.002792 / 0.011008 (-0.008216) | 0.048027 / 0.038508 (0.009519) | 0.059256 / 0.023109 (0.036147) | 0.270150 / 0.275898 (-0.005748) | 0.294530 / 0.323480 (-0.028950) | 0.004162 / 0.007986 (-0.003823) | 0.002470 / 0.004328 (-0.001858) | 0.047993 / 0.004250 (0.043743) | 0.040380 / 0.037052 (0.003328) | 0.275247 / 0.258489 (0.016758) | 0.305684 / 0.293841 (0.011843) | 0.025072 / 0.128546 (-0.103474) | 0.007183 / 0.075646 (-0.068463) | 0.054875 / 0.419271 (-0.364397) | 0.033053 / 0.043533 (-0.010480) | 0.271281 / 0.255139 (0.016142) | 0.288057 / 0.283200 (0.004858) | 0.018692 / 0.141683 (-0.122991) | 1.125224 / 1.452155 (-0.326930) | 1.171083 / 1.492716 (-0.321633) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.103102 / 0.018006 (0.085096) | 0.309099 / 0.000490 (0.308609) | 0.000232 / 0.000200 (0.000032) | 0.000052 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021532 / 0.037411 (-0.015879) | 0.069927 / 0.014526 (0.055401) | 0.080920 / 0.176557 (-0.095637) | 0.122214 / 0.737135 (-0.614921) | 0.082268 / 0.296338 (-0.214071) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.298121 / 0.215209 (0.082912) | 2.933000 / 2.077655 (0.855345) | 1.608782 / 1.504120 (0.104662) | 1.554083 / 1.541195 (0.012889) | 1.552700 / 1.468490 (0.084209) | 0.400576 / 4.584777 (-4.184201) | 2.412914 / 3.745712 (-1.332798) | 2.545706 / 5.269862 (-2.724155) | 1.548797 / 4.565676 (-3.016879) | 0.045553 / 0.424275 (-0.378722) | 0.004751 / 0.007607 (-0.002857) | 0.343002 / 0.226044 (0.116958) | 3.402866 / 2.268929 (1.133937) | 1.969910 / 55.444624 (-53.474715) | 1.686639 / 6.876477 (-5.189838) | 1.768474 / 2.142072 (-0.373599) | 0.471299 / 4.805227 (-4.333928) | 0.097696 / 6.500664 (-6.402968) | 0.041693 / 0.075469 (-0.033776) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.971380 / 1.841788 (-0.870408) | 12.686033 / 8.074308 (4.611725) | 11.370946 / 10.191392 (1.179554) | 0.138377 / 0.680424 (-0.542047) | 0.015623 / 0.534201 (-0.518578) | 0.270935 / 0.579283 (-0.308348) | 0.276235 / 0.434364 (-0.158129) | 0.310196 / 0.540337 (-0.230141) | 0.416908 / 1.386936 (-0.970028) |

</details>
</details>

![](https://cml.dev/watermark.png#bf02cff8d70180a9e89328961ded9e3d8510fd22 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004581 / 0.011353 (-0.006772) | 0.002468 / 0.011008 (-0.008541) | 0.061420 / 0.038508 (0.022912) | 0.047685 / 0.023109 (0.024575) | 0.237756 / 0.275898 (-0.038142) | 0.267548 / 0.323480 (-0.055932) | 0.003899 / 0.007986 (-0.004086) | 0.002338 / 0.004328 (-0.001990) | 0.048794 / 0.004250 (0.044543) | 0.042485 / 0.037052 (0.005433) | 0.250165 / 0.258489 (-0.008324) | 0.278791 / 0.293841 (-0.015050) | 0.022371 / 0.128546 (-0.106175) | 0.006923 / 0.075646 (-0.068723) | 0.201401 / 0.419271 (-0.217870) | 0.035867 / 0.043533 (-0.007665) | 0.244628 / 0.255139 (-0.010511) | 0.271137 / 0.283200 (-0.012063) | 0.017257 / 0.141683 (-0.124426) | 1.097261 / 1.452155 (-0.354894) | 1.163314 / 1.492716 (-0.329402) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.089060 / 0.018006 (0.071054) | 0.297489 / 0.000490 (0.296999) | 0.000207 / 0.000200 (0.000007) | 0.000050 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018583 / 0.037411 (-0.018828) | 0.061974 / 0.014526 (0.047449) | 0.073300 / 0.176557 (-0.103256) | 0.118871 / 0.737135 (-0.618264) | 0.075513 / 0.296338 (-0.220826) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.285544 / 0.215209 (0.070335) | 2.799871 / 2.077655 (0.722216) | 1.479871 / 1.504120 (-0.024249) | 1.351128 / 1.541195 (-0.190067) | 1.377540 / 1.468490 (-0.090950) | 0.393056 / 4.584777 (-4.191721) | 2.341791 / 3.745712 (-1.403921) | 2.546854 / 5.269862 (-2.723007) | 1.547368 / 4.565676 (-3.018309) | 0.046056 / 0.424275 (-0.378219) | 0.004765 / 0.007607 (-0.002842) | 0.336384 / 0.226044 (0.110339) | 3.283277 / 2.268929 (1.014348) | 1.784535 / 55.444624 (-53.660089) | 1.557809 / 6.876477 (-5.318667) | 1.581728 / 2.142072 (-0.560344) | 0.470527 / 4.805227 (-4.334700) | 0.098383 / 6.500664 (-6.402281) | 0.041563 / 0.075469 (-0.033906) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.946924 / 1.841788 (-0.894863) | 11.202775 / 8.074308 (3.128467) | 10.249760 / 10.191392 (0.058368) | 0.142337 / 0.680424 (-0.538087) | 0.013784 / 0.534201 (-0.520417) | 0.267237 / 0.579283 (-0.312046) | 0.264142 / 0.434364 (-0.170222) | 0.306343 / 0.540337 (-0.233994) | 0.423681 / 1.386936 (-0.963255) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004786 / 0.011353 (-0.006567) | 0.002398 / 0.011008 (-0.008610) | 0.047325 / 0.038508 (0.008817) | 0.050753 / 0.023109 (0.027644) | 0.271132 / 0.275898 (-0.004766) | 0.290854 / 0.323480 (-0.032626) | 0.003953 / 0.007986 (-0.004033) | 0.002238 / 0.004328 (-0.002090) | 0.047463 / 0.004250 (0.043213) | 0.038504 / 0.037052 (0.001451) | 0.273182 / 0.258489 (0.014693) | 0.303449 / 0.293841 (0.009608) | 0.024069 / 0.128546 (-0.104477) | 0.006712 / 0.075646 (-0.068934) | 0.053032 / 0.419271 (-0.366239) | 0.032221 / 0.043533 (-0.011312) | 0.271770 / 0.255139 (0.016631) | 0.287876 / 0.283200 (0.004677) | 0.018040 / 0.141683 (-0.123643) | 1.138749 / 1.452155 (-0.313405) | 1.192048 / 1.492716 (-0.300668) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.089132 / 0.018006 (0.071126) | 0.298636 / 0.000490 (0.298146) | 0.000220 / 0.000200 (0.000020) | 0.000053 / 0.000054 (-0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.020808 / 0.037411 (-0.016603) | 0.069506 / 0.014526 (0.054980) | 0.079412 / 0.176557 (-0.097145) | 0.118188 / 0.737135 (-0.618947) | 0.083044 / 0.296338 (-0.213294) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.293502 / 0.215209 (0.078293) | 2.863692 / 2.077655 (0.786037) | 1.590877 / 1.504120 (0.086757) | 1.483634 / 1.541195 (-0.057561) | 1.502113 / 1.468490 (0.033623) | 0.402170 / 4.584777 (-4.182607) | 2.414188 / 3.745712 (-1.331524) | 2.500146 / 5.269862 (-2.769716) | 1.506977 / 4.565676 (-3.058699) | 0.045849 / 0.424275 (-0.378426) | 0.004755 / 0.007607 (-0.002852) | 0.343073 / 0.226044 (0.117029) | 3.354985 / 2.268929 (1.086056) | 1.952594 / 55.444624 (-53.492030) | 1.664084 / 6.876477 (-5.212392) | 1.664203 / 2.142072 (-0.477869) | 0.475858 / 4.805227 (-4.329370) | 0.097539 / 6.500664 (-6.403125) | 0.040201 / 0.075469 (-0.035268) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.980051 / 1.841788 (-0.861736) | 11.615291 / 8.074308 (3.540983) | 10.492092 / 10.191392 (0.300700) | 0.130450 / 0.680424 (-0.549974) | 0.015883 / 0.534201 (-0.518318) | 0.267575 / 0.579283 (-0.311708) | 0.276981 / 0.434364 (-0.157383) | 0.310221 / 0.540337 (-0.230116) | 0.417143 / 1.386936 (-0.969793) |

</details>
</details>

![](https://cml.dev/watermark.png#bf02cff8d70180a9e89328961ded9e3d8510fd22 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004721 / 0.011353 (-0.006632) | 0.002931 / 0.011008 (-0.008077) | 0.061948 / 0.038508 (0.023440) | 0.051066 / 0.023109 (0.027957) | 0.245431 / 0.275898 (-0.030467) | 0.295627 / 0.323480 (-0.027852) | 0.003997 / 0.007986 (-0.003988) | 0.002408 / 0.004328 (-0.001920) | 0.048292 / 0.004250 (0.044041) | 0.044716 / 0.037052 (0.007664) | 0.255119 / 0.258489 (-0.003371) | 0.287384 / 0.293841 (-0.006457) | 0.022835 / 0.128546 (-0.105711) | 0.007162 / 0.075646 (-0.068484) | 0.201352 / 0.419271 (-0.217920) | 0.036626 / 0.043533 (-0.006906) | 0.249590 / 0.255139 (-0.005549) | 0.270822 / 0.283200 (-0.012378) | 0.018152 / 0.141683 (-0.123531) | 1.097046 / 1.452155 (-0.355109) | 1.160461 / 1.492716 (-0.332255) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091712 / 0.018006 (0.073705) | 0.299121 / 0.000490 (0.298631) | 0.000244 / 0.000200 (0.000044) | 0.000055 / 0.000054 (0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018998 / 0.037411 (-0.018413) | 0.062811 / 0.014526 (0.048285) | 0.076348 / 0.176557 (-0.100209) | 0.123898 / 0.737135 (-0.613238) | 0.076249 / 0.296338 (-0.220090) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.282780 / 0.215209 (0.067571) | 2.739028 / 2.077655 (0.661373) | 1.472564 / 1.504120 (-0.031556) | 1.347343 / 1.541195 (-0.193852) | 1.387130 / 1.468490 (-0.081360) | 0.403348 / 4.584777 (-4.181429) | 2.369924 / 3.745712 (-1.375788) | 2.612875 / 5.269862 (-2.656987) | 1.588079 / 4.565676 (-2.977598) | 0.045233 / 0.424275 (-0.379042) | 0.004767 / 0.007607 (-0.002840) | 0.336614 / 0.226044 (0.110570) | 3.300485 / 2.268929 (1.031556) | 1.834365 / 55.444624 (-53.610259) | 1.559799 / 6.876477 (-5.316677) | 1.601265 / 2.142072 (-0.540808) | 0.468158 / 4.805227 (-4.337069) | 0.099811 / 6.500664 (-6.400853) | 0.042688 / 0.075469 (-0.032782) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.934097 / 1.841788 (-0.907691) | 11.687713 / 8.074308 (3.613405) | 10.412723 / 10.191392 (0.221331) | 0.139276 / 0.680424 (-0.541148) | 0.014042 / 0.534201 (-0.520159) | 0.270306 / 0.579283 (-0.308978) | 0.266609 / 0.434364 (-0.167755) | 0.314179 / 0.540337 (-0.226158) | 0.437744 / 1.386936 (-0.949192) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004893 / 0.011353 (-0.006460) | 0.002952 / 0.011008 (-0.008056) | 0.050441 / 0.038508 (0.011933) | 0.051838 / 0.023109 (0.028729) | 0.271163 / 0.275898 (-0.004735) | 0.293031 / 0.323480 (-0.030449) | 0.003976 / 0.007986 (-0.004010) | 0.002396 / 0.004328 (-0.001933) | 0.048103 / 0.004250 (0.043852) | 0.038732 / 0.037052 (0.001680) | 0.274276 / 0.258489 (0.015787) | 0.305112 / 0.293841 (0.011271) | 0.024112 / 0.128546 (-0.104434) | 0.007203 / 0.075646 (-0.068443) | 0.053502 / 0.419271 (-0.365770) | 0.032360 / 0.043533 (-0.011173) | 0.270154 / 0.255139 (0.015015) | 0.286689 / 0.283200 (0.003489) | 0.018285 / 0.141683 (-0.123397) | 1.141421 / 1.452155 (-0.310734) | 1.244062 / 1.492716 (-0.248654) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090960 / 0.018006 (0.072954) | 0.286134 / 0.000490 (0.285644) | 0.000207 / 0.000200 (0.000007) | 0.000045 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.020789 / 0.037411 (-0.016622) | 0.070850 / 0.014526 (0.056324) | 0.080750 / 0.176557 (-0.095807) | 0.120046 / 0.737135 (-0.617089) | 0.083630 / 0.296338 (-0.212708) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.290654 / 0.215209 (0.075445) | 2.846669 / 2.077655 (0.769014) | 1.561752 / 1.504120 (0.057632) | 1.442968 / 1.541195 (-0.098227) | 1.503551 / 1.468490 (0.035061) | 0.399731 / 4.584777 (-4.185046) | 2.430099 / 3.745712 (-1.315613) | 2.556169 / 5.269862 (-2.713692) | 1.545591 / 4.565676 (-3.020085) | 0.045967 / 0.424275 (-0.378309) | 0.004851 / 0.007607 (-0.002756) | 0.340167 / 0.226044 (0.114122) | 3.392738 / 2.268929 (1.123809) | 1.943577 / 55.444624 (-53.501047) | 1.650057 / 6.876477 (-5.226420) | 1.686872 / 2.142072 (-0.455201) | 0.470305 / 4.805227 (-4.334923) | 0.097296 / 6.500664 (-6.403368) | 0.041399 / 0.075469 (-0.034070) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.985660 / 1.841788 (-0.856128) | 12.300826 / 8.074308 (4.226518) | 10.972591 / 10.191392 (0.781199) | 0.131512 / 0.680424 (-0.548912) | 0.015742 / 0.534201 (-0.518459) | 0.270630 / 0.579283 (-0.308653) | 0.276039 / 0.434364 (-0.158325) | 0.302288 / 0.540337 (-0.238050) | 0.409415 / 1.386936 (-0.977521) |

</details>
</details>

![](https://cml.dev/watermark.png#bf02cff8d70180a9e89328961ded9e3d8510fd22 "CML watermark")