Fix ArrowExamplesIterable.shard_data_sources
Opened by lhoestq (closed)

ArrowExamplesIterable.shard_data_sources was outdated

I also fixed a warning message by not using format_type= in with_format()

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005893 / 0.011353 (-0.005460) | 0.003682 / 0.011008 (-0.007327) | 0.098358 / 0.038508 (0.059850) | 0.028130 / 0.023109 (0.005020) | 0.305960 / 0.275898 (0.030062) | 0.334869 / 0.323480 (0.011390) | 0.003522 / 0.007986 (-0.004463) | 0.003683 / 0.004328 (-0.000645) | 0.079418 / 0.004250 (0.075168) | 0.037662 / 0.037052 (0.000609) | 0.310893 / 0.258489 (0.052404) | 0.341347 / 0.293841 (0.047506) | 0.027450 / 0.128546 (-0.101096) | 0.008381 / 0.075646 (-0.067265) | 0.316020 / 0.419271 (-0.103252) | 0.045079 / 0.043533 (0.001546) | 0.307806 / 0.255139 (0.052667) | 0.331804 / 0.283200 (0.048604) | 0.091806 / 0.141683 (-0.049877) | 1.492611 / 1.452155 (0.040457) | 1.551762 / 1.492716 (0.059046) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.201640 / 0.018006 (0.183634) | 0.422776 / 0.000490 (0.422286) | 0.003734 / 0.000200 (0.003535) | 0.000080 / 0.000054 (0.000026) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025429 / 0.037411 (-0.011982) | 0.104699 / 0.014526 (0.090173) | 0.110505 / 0.176557 (-0.066051) | 0.171252 / 0.737135 (-0.565883) | 0.113131 / 0.296338 (-0.183208) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.419914 / 0.215209 (0.204705) | 4.184414 / 2.077655 (2.106760) | 1.999263 / 1.504120 (0.495143) | 1.828669 / 1.541195 (0.287474) | 1.940366 / 1.468490 (0.471876) | 0.556939 / 4.584777 (-4.027838) | 3.389164 / 3.745712 (-0.356548) | 1.796323 / 5.269862 (-3.473538) | 1.048843 / 4.565676 (-3.516833) | 0.067315 / 0.424275 (-0.356960) | 0.011531 / 0.007607 (0.003923) | 0.517226 / 0.226044 (0.291182) | 5.167255 / 2.268929 (2.898326) | 2.431129 / 55.444624 (-53.013495) | 2.133913 / 6.876477 (-4.742564) | 2.359021 / 2.142072 (0.216948) | 0.666390 / 4.805227 (-4.138838) | 0.135147 / 6.500664 (-6.365517) | 0.064855 / 0.075469 (-0.010614) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.166530 / 1.841788 (-0.675258) | 14.060551 / 8.074308 (5.986242) | 14.171663 / 10.191392 (3.980271) | 0.285821 / 0.680424 (-0.394603) | 0.016867 / 0.534201 (-0.517334) | 0.369102 / 0.579283 (-0.210181) | 0.393580 / 0.434364 (-0.040784) | 0.423721 / 0.540337 (-0.116616) | 0.512559 / 1.386936 (-0.874377) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006674 / 0.011353 (-0.004679) | 0.004006 / 0.011008 (-0.007002) | 0.080160 / 0.038508 (0.041652) | 0.032508 / 0.023109 (0.009399) | 0.378168 / 0.275898 (0.102270) | 0.417796 / 0.323480 (0.094316) | 0.003706 / 0.007986 (-0.004280) | 0.002995 / 0.004328 (-0.001333) | 0.079275 / 0.004250 (0.075025) | 0.043690 / 0.037052 (0.006638) | 0.377717 / 0.258489 (0.119228) | 0.439801 / 0.293841 (0.145961) | 0.028438 / 0.128546 (-0.100108) | 0.008661 / 0.075646 (-0.066985) | 0.085280 / 0.419271 (-0.333991) | 0.043716 / 0.043533 (0.000183) | 0.370086 / 0.255139 (0.114947) | 0.403763 / 0.283200 (0.120563) | 0.095022 / 0.141683 (-0.046661) | 1.534376 / 1.452155 (0.082221) | 1.597658 / 1.492716 (0.104942) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.240229 / 0.018006 (0.222223) | 0.496281 / 0.000490 (0.495792) | 0.002165 / 0.000200 (0.001965) | 0.000075 / 0.000054 (0.000020) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025330 / 0.037411 (-0.012081) | 0.102414 / 0.014526 (0.087888) | 0.112733 / 0.176557 (-0.063824) | 0.161181 / 0.737135 (-0.575955) | 0.114196 / 0.296338 (-0.182143) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.456808 / 0.215209 (0.241599) | 4.534937 / 2.077655 (2.457283) | 2.318834 / 1.504120 (0.814714) | 2.074085 / 1.541195 (0.532890) | 2.117409 / 1.468490 (0.648919) | 0.559110 / 4.584777 (-4.025667) | 3.371695 / 3.745712 (-0.374017) | 2.543154 / 5.269862 (-2.726708) | 1.360552 / 4.565676 (-3.205125) | 0.067602 / 0.424275 (-0.356674) | 0.011396 / 0.007607 (0.003789) | 0.561666 / 0.226044 (0.335622) | 5.607666 / 2.268929 (3.338737) | 2.802775 / 55.444624 (-52.641849) | 2.486162 / 6.876477 (-4.390315) | 2.390885 / 2.142072 (0.248813) | 0.667407 / 4.805227 (-4.137820) | 0.135948 / 6.500664 (-6.364717) | 0.067272 / 0.075469 (-0.008197) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.279664 / 1.841788 (-0.562124) | 15.188099 / 8.074308 (7.113791) | 14.380355 / 10.191392 (4.188963) | 0.140344 / 0.680424 (-0.540080) | 0.016832 / 0.534201 (-0.517369) | 0.364631 / 0.579283 (-0.214652) | 0.400306 / 0.434364 (-0.034058) | 0.430793 / 0.540337 (-0.109545) | 0.525923 / 1.386936 (-0.861013) |

</details>
</details>

![](https://cml.dev/watermark.png#48ca19cf1f4d1c99765a1f847c1f6b849496d99d "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008502 / 0.011353 (-0.002851) | 0.005946 / 0.011008 (-0.005062) | 0.131279 / 0.038508 (0.092771) | 0.035400 / 0.023109 (0.012291) | 0.423240 / 0.275898 (0.147342) | 0.470248 / 0.323480 (0.146768) | 0.004949 / 0.007986 (-0.003037) | 0.004544 / 0.004328 (0.000215) | 0.106856 / 0.004250 (0.102605) | 0.046579 / 0.037052 (0.009527) | 0.441135 / 0.258489 (0.182646) | 0.470401 / 0.293841 (0.176561) | 0.047231 / 0.128546 (-0.081315) | 0.017278 / 0.075646 (-0.058368) | 0.401937 / 0.419271 (-0.017335) | 0.067151 / 0.043533 (0.023619) | 0.453908 / 0.255139 (0.198769) | 0.422171 / 0.283200 (0.138971) | 0.123583 / 0.141683 (-0.018100) | 1.852895 / 1.452155 (0.400740) | 1.827282 / 1.492716 (0.334566) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.246419 / 0.018006 (0.228413) | 0.576930 / 0.000490 (0.576440) | 0.007511 / 0.000200 (0.007312) | 0.000165 / 0.000054 (0.000111) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032732 / 0.037411 (-0.004680) | 0.130266 / 0.014526 (0.115740) | 0.150537 / 0.176557 (-0.026019) | 0.218554 / 0.737135 (-0.518582) | 0.148572 / 0.296338 (-0.147766) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.598611 / 0.215209 (0.383402) | 6.181219 / 2.077655 (4.103564) | 2.473468 / 1.504120 (0.969348) | 2.206374 / 1.541195 (0.665179) | 2.216707 / 1.468490 (0.748217) | 0.981295 / 4.584777 (-3.603482) | 5.716384 / 3.745712 (1.970672) | 5.882327 / 5.269862 (0.612466) | 2.761081 / 4.565676 (-1.804595) | 0.113544 / 0.424275 (-0.310731) | 0.015131 / 0.007607 (0.007524) | 0.850939 / 0.226044 (0.624894) | 8.046611 / 2.268929 (5.777682) | 3.340542 / 55.444624 (-52.104083) | 2.673692 / 6.876477 (-4.202785) | 2.926330 / 2.142072 (0.784257) | 1.176164 / 4.805227 (-3.629064) | 0.226745 / 6.500664 (-6.273919) | 0.085910 / 0.075469 (0.010441) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.483792 / 1.841788 (-0.357995) | 18.895009 / 8.074308 (10.820701) | 20.982461 / 10.191392 (10.791069) | 0.253085 / 0.680424 (-0.427339) | 0.031284 / 0.534201 (-0.502917) | 0.516569 / 0.579283 (-0.062714) | 0.635781 / 0.434364 (0.201417) | 0.604359 / 0.540337 (0.064022) | 0.725278 / 1.386936 (-0.661658) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009220 / 0.011353 (-0.002133) | 0.005792 / 0.011008 (-0.005216) | 0.099795 / 0.038508 (0.061287) | 0.033812 / 0.023109 (0.010703) | 0.459386 / 0.275898 (0.183488) | 0.518067 / 0.323480 (0.194587) | 0.005083 / 0.007986 (-0.002902) | 0.004145 / 0.004328 (-0.000183) | 0.103506 / 0.004250 (0.099255) | 0.050429 / 0.037052 (0.013377) | 0.478149 / 0.258489 (0.219660) | 0.531280 / 0.293841 (0.237440) | 0.047373 / 0.128546 (-0.081173) | 0.013647 / 0.075646 (-0.061999) | 0.115174 / 0.419271 (-0.304098) | 0.061099 / 0.043533 (0.017566) | 0.455002 / 0.255139 (0.199863) | 0.507765 / 0.283200 (0.224565) | 0.112219 / 0.141683 (-0.029464) | 1.873591 / 1.452155 (0.421436) | 1.952061 / 1.492716 (0.459345) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.283587 / 0.018006 (0.265581) | 0.587562 / 0.000490 (0.587073) | 0.001252 / 0.000200 (0.001052) | 0.000095 / 0.000054 (0.000040) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032706 / 0.037411 (-0.004705) | 0.137715 / 0.014526 (0.123189) | 0.131932 / 0.176557 (-0.044625) | 0.200042 / 0.737135 (-0.537094) | 0.159327 / 0.296338 (-0.137011) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.624061 / 0.215209 (0.408852) | 6.386235 / 2.077655 (4.308580) | 2.908786 / 1.504120 (1.404666) | 2.589855 / 1.541195 (1.048660) | 2.387988 / 1.468490 (0.919498) | 0.952625 / 4.584777 (-3.632152) | 5.571641 / 3.745712 (1.825929) | 2.711154 / 5.269862 (-2.558708) | 1.788015 / 4.565676 (-2.777662) | 0.104488 / 0.424275 (-0.319787) | 0.015213 / 0.007607 (0.007606) | 0.798446 / 0.226044 (0.572401) | 8.011614 / 2.268929 (5.742686) | 3.711951 / 55.444624 (-51.732673) | 2.896881 / 6.876477 (-3.979595) | 3.172116 / 2.142072 (1.030043) | 1.136816 / 4.805227 (-3.668411) | 0.239254 / 6.500664 (-6.261410) | 0.081136 / 0.075469 (0.005667) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.798246 / 1.841788 (-0.043542) | 19.497108 / 8.074308 (11.422800) | 23.450258 / 10.191392 (13.258866) | 0.250021 / 0.680424 (-0.430403) | 0.029138 / 0.534201 (-0.505063) | 0.532984 / 0.579283 (-0.046299) | 0.638161 / 0.434364 (0.203797) | 0.615720 / 0.540337 (0.075382) | 0.770621 / 1.386936 (-0.616315) |

</details>
</details>

![](https://cml.dev/watermark.png#7d8345c5f8a844ff44cfbb30cbda514ffe89bfd7 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009120 / 0.011353 (-0.002233) | 0.005381 / 0.011008 (-0.005627) | 0.139719 / 0.038508 (0.101211) | 0.037229 / 0.023109 (0.014120) | 0.414633 / 0.275898 (0.138734) | 0.480313 / 0.323480 (0.156833) | 0.005027 / 0.007986 (-0.002959) | 0.005015 / 0.004328 (0.000687) | 0.108513 / 0.004250 (0.104263) | 0.056167 / 0.037052 (0.019115) | 0.407588 / 0.258489 (0.149099) | 0.518899 / 0.293841 (0.225058) | 0.048857 / 0.128546 (-0.079689) | 0.013694 / 0.075646 (-0.061952) | 0.418035 / 0.419271 (-0.001237) | 0.067755 / 0.043533 (0.024222) | 0.417740 / 0.255139 (0.162601) | 0.478622 / 0.283200 (0.195422) | 0.118290 / 0.141683 (-0.023393) | 1.901473 / 1.452155 (0.449319) | 1.978126 / 1.492716 (0.485409) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.271960 / 0.018006 (0.253954) | 0.602745 / 0.000490 (0.602255) | 0.005371 / 0.000200 (0.005171) | 0.000102 / 0.000054 (0.000048) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029620 / 0.037411 (-0.007791) | 0.122402 / 0.014526 (0.107877) | 0.132645 / 0.176557 (-0.043911) | 0.212635 / 0.737135 (-0.524500) | 0.136901 / 0.296338 (-0.159438) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.644017 / 0.215209 (0.428808) | 6.597151 / 2.077655 (4.519496) | 2.454471 / 1.504120 (0.950351) | 2.151357 / 1.541195 (0.610163) | 2.290748 / 1.468490 (0.822258) | 0.970194 / 4.584777 (-3.614583) | 5.475275 / 3.745712 (1.729563) | 2.772658 / 5.269862 (-2.497204) | 1.785311 / 4.565676 (-2.780366) | 0.114503 / 0.424275 (-0.309772) | 0.015374 / 0.007607 (0.007767) | 0.768413 / 0.226044 (0.542368) | 7.956219 / 2.268929 (5.687290) | 3.272138 / 55.444624 (-52.172486) | 2.539638 / 6.876477 (-4.336839) | 2.713526 / 2.142072 (0.571454) | 1.181221 / 4.805227 (-3.624006) | 0.236327 / 6.500664 (-6.264337) | 0.089815 / 0.075469 (0.014345) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.521805 / 1.841788 (-0.319983) | 18.196529 / 8.074308 (10.122221) | 20.287324 / 10.191392 (10.095932) | 0.256959 / 0.680424 (-0.423465) | 0.028846 / 0.534201 (-0.505355) | 0.522354 / 0.579283 (-0.056929) | 0.600216 / 0.434364 (0.165852) | 0.607668 / 0.540337 (0.067331) | 0.762101 / 1.386936 (-0.624835) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009227 / 0.011353 (-0.002126) | 0.005398 / 0.011008 (-0.005610) | 0.094998 / 0.038508 (0.056490) | 0.036633 / 0.023109 (0.013524) | 0.493317 / 0.275898 (0.217419) | 0.517216 / 0.323480 (0.193736) | 0.005510 / 0.007986 (-0.002476) | 0.004249 / 0.004328 (-0.000079) | 0.107936 / 0.004250 (0.103685) | 0.050223 / 0.037052 (0.013171) | 0.580275 / 0.258489 (0.321786) | 0.551477 / 0.293841 (0.257636) | 0.048758 / 0.128546 (-0.079788) | 0.013954 / 0.075646 (-0.061692) | 0.107021 / 0.419271 (-0.312250) | 0.064416 / 0.043533 (0.020884) | 0.485225 / 0.255139 (0.230086) | 0.513862 / 0.283200 (0.230663) | 0.118848 / 0.141683 (-0.022835) | 1.755396 / 1.452155 (0.303241) | 1.970349 / 1.492716 (0.477633) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.290743 / 0.018006 (0.272737) | 0.603293 / 0.000490 (0.602803) | 0.006814 / 0.000200 (0.006614) | 0.000156 / 0.000054 (0.000101) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029862 / 0.037411 (-0.007550) | 0.136530 / 0.014526 (0.122005) | 0.133728 / 0.176557 (-0.042829) | 0.194709 / 0.737135 (-0.542427) | 0.151080 / 0.296338 (-0.145258) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.649202 / 0.215209 (0.433993) | 6.637578 / 2.077655 (4.559923) | 3.040135 / 1.504120 (1.536015) | 2.671308 / 1.541195 (1.130113) | 2.722412 / 1.468490 (1.253922) | 0.953029 / 4.584777 (-3.631748) | 5.805002 / 3.745712 (2.059290) | 5.049939 / 5.269862 (-0.219922) | 2.284053 / 4.565676 (-2.281623) | 0.130399 / 0.424275 (-0.293876) | 0.014726 / 0.007607 (0.007119) | 0.932570 / 0.226044 (0.706526) | 8.576693 / 2.268929 (6.307765) | 4.032738 / 55.444624 (-51.411886) | 3.274715 / 6.876477 (-3.601762) | 3.513788 / 2.142072 (1.371716) | 1.130624 / 4.805227 (-3.674603) | 0.219597 / 6.500664 (-6.281067) | 0.081425 / 0.075469 (0.005956) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.735312 / 1.841788 (-0.106476) | 18.438587 / 8.074308 (10.364279) | 21.582310 / 10.191392 (11.390918) | 0.224040 / 0.680424 (-0.456384) | 0.027590 / 0.534201 (-0.506611) | 0.503598 / 0.579283 (-0.075685) | 0.624379 / 0.434364 (0.190015) | 0.571911 / 0.540337 (0.031574) | 0.723215 / 1.386936 (-0.663721) |

</details>
</details>

![](https://cml.dev/watermark.png#9e40d28f2b0060a429c70827191fa5ff3ce8cf27 "CML watermark")