Fix map suffix_template
State: closed | Author: lhoestq (MEMBER)

#5455 introduced a small bug that lead `map` to ignore the `suffix_template` argument and not put suffixes to cached files in multiprocessing.

I fixed this and also improved a few things:

- regarding logging: "Loading cached processed dataset" is now logged only once even in multiprocessing (it used to be logged `num_proc` times)
- regarding new_fingerprint: I made sure that the returned dataset satisfies `ds._fingerprint==new_fingerprint` if `new_fingerprint` is passed to `map`

Comments: _The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.011596 / 0.011353 (0.000244) | 0.005845 / 0.011008 (-0.005164) | 0.121302 / 0.038508 (0.082794) | 0.034306 / 0.023109 (0.011196) | 0.355973 / 0.275898 (0.080075) | 0.419903 / 0.323480 (0.096423) | 0.009049 / 0.007986 (0.001064) | 0.004245 / 0.004328 (-0.000084) | 0.092004 / 0.004250 (0.087753) | 0.042782 / 0.037052 (0.005730) | 0.355805 / 0.258489 (0.097316) | 0.407298 / 0.293841 (0.113457) | 0.052481 / 0.128546 (-0.076066) | 0.020880 / 0.075646 (-0.054766) | 0.379948 / 0.419271 (-0.039324) | 0.061337 / 0.043533 (0.017804) | 0.359829 / 0.255139 (0.104690) | 0.379244 / 0.283200 (0.096044) | 0.116692 / 0.141683 (-0.024990) | 1.733717 / 1.452155 (0.281562) | 1.700246 / 1.492716 (0.207530) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.014622 / 0.018006 (-0.003384) | 0.518777 / 0.000490 (0.518288) | 0.004086 / 0.000200 (0.003886) | 0.000136 / 0.000054 (0.000082) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031208 / 0.037411 (-0.006204) | 0.143003 / 0.014526 (0.128477) | 0.132625 / 0.176557 (-0.043932) | 0.187681 / 0.737135 (-0.549455) | 0.136576 / 0.296338 (-0.159763) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.626516 / 0.215209 (0.411307) | 6.282558 / 2.077655 (4.204904) | 2.702686 / 1.504120 (1.198566) | 2.287445 / 1.541195 (0.746250) | 2.333014 / 1.468490 (0.864524) | 1.227815 / 4.584777 (-3.356962) | 5.545640 / 3.745712 (1.799928) | 4.953226 / 5.269862 (-0.316635) | 2.774549 / 4.565676 (-1.791128) | 0.145257 / 0.424275 (-0.279018) | 0.014887 / 0.007607 (0.007280) | 0.812226 / 0.226044 (0.586182) | 8.002727 / 2.268929 (5.733798) | 3.314852 / 55.444624 (-52.129773) | 2.602348 / 6.876477 (-4.274128) | 2.593511 / 2.142072 (0.451438) | 1.440498 / 4.805227 (-3.364730) | 0.254849 / 6.500664 (-6.245815) | 0.077020 / 0.075469 (0.001551) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.487633 / 1.841788 (-0.354155) | 17.385773 / 8.074308 (9.311465) | 21.775511 / 10.191392 (11.584118) | 0.273514 / 0.680424 (-0.406910) | 0.059644 / 0.534201 (-0.474557) | 0.578710 / 0.579283 (-0.000573) | 0.630221 / 0.434364 (0.195857) | 0.632089 / 0.540337 (0.091752) | 0.762367 / 1.386936 (-0.624569) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009513 / 0.011353 (-0.001840) | 0.006009 / 0.011008 (-0.004999) | 0.087589 / 0.038508 (0.049081) | 0.037487 / 0.023109 (0.014378) | 0.397660 / 0.275898 (0.121762) | 0.474438 / 0.323480 (0.150958) | 0.007373 / 0.007986 (-0.000613) | 0.005839 / 0.004328 (0.001511) | 0.092759 / 0.004250 (0.088509) | 0.052128 / 0.037052 (0.015075) | 0.382378 / 0.258489 (0.123889) | 0.458244 / 0.293841 (0.164403) | 0.057232 / 0.128546 (-0.071314) | 0.020662 / 0.075646 (-0.054984) | 0.110314 / 0.419271 (-0.308957) | 0.063014 / 0.043533 (0.019481) | 0.386020 / 0.255139 (0.130881) | 0.476169 / 0.283200 (0.192970) | 0.118081 / 0.141683 (-0.023602) | 1.724158 / 1.452155 (0.272003) | 1.862257 / 1.492716 (0.369541) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.224288 / 0.018006 (0.206281) | 0.523631 / 0.000490 (0.523141) | 0.004420 / 0.000200 (0.004220) | 0.000127 / 0.000054 (0.000073) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032359 / 0.037411 (-0.005052) | 0.140045 / 0.014526 (0.125519) | 0.138164 / 0.176557 (-0.038393) | 0.181068 / 0.737135 (-0.556067) | 0.143965 / 0.296338 (-0.152374) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.573809 / 0.215209 (0.358600) | 6.083247 / 2.077655 (4.005592) | 2.671258 / 1.504120 (1.167138) | 2.277062 / 1.541195 (0.735868) | 2.299544 / 1.468490 (0.831054) | 1.267351 / 4.584777 (-3.317425) | 5.494461 / 3.745712 (1.748749) | 5.083169 / 5.269862 (-0.186692) | 2.531738 / 4.565676 (-2.033938) | 0.151834 / 0.424275 (-0.272441) | 0.014123 / 0.007607 (0.006516) | 0.800222 / 0.226044 (0.574177) | 7.637624 / 2.268929 (5.368695) | 3.325574 / 55.444624 (-52.119050) | 2.563008 / 6.876477 (-4.313468) | 2.596259 / 2.142072 (0.454187) | 1.459206 / 4.805227 (-3.346021) | 0.237771 / 6.500664 (-6.262893) | 0.071854 / 0.075469 (-0.003615) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.605504 / 1.841788 (-0.236284) | 17.593594 / 8.074308 (9.519285) | 20.618005 / 10.191392 (10.426612) | 0.270938 / 0.680424 (-0.409486) | 0.026205 / 0.534201 (-0.507996) | 0.562223 / 0.579283 (-0.017060) | 0.617571 / 0.434364 (0.183207) | 0.616398 / 0.540337 (0.076060) | 0.715293 / 1.386936 (-0.671643) |

</details>
</details>

![](https://cml.dev/watermark.png#673dc0dd7d063b2313f7adcc9e0be53d4718f5cf "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.013213 / 0.011353 (0.001860) | 0.006253 / 0.011008 (-0.004756) | 0.125175 / 0.038508 (0.086667) | 0.037491 / 0.023109 (0.014382) | 0.401379 / 0.275898 (0.125481) | 0.395826 / 0.323480 (0.072346) | 0.009224 / 0.007986 (0.001238) | 0.005163 / 0.004328 (0.000835) | 0.096490 / 0.004250 (0.092239) | 0.042473 / 0.037052 (0.005420) | 0.383713 / 0.258489 (0.125224) | 0.429234 / 0.293841 (0.135393) | 0.063261 / 0.128546 (-0.065285) | 0.020114 / 0.075646 (-0.055532) | 0.401687 / 0.419271 (-0.017585) | 0.062831 / 0.043533 (0.019298) | 0.405211 / 0.255139 (0.150072) | 0.380810 / 0.283200 (0.097610) | 0.109166 / 0.141683 (-0.032517) | 1.869580 / 1.452155 (0.417426) | 1.949947 / 1.492716 (0.457231) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.207481 / 0.018006 (0.189475) | 0.504161 / 0.000490 (0.503671) | 0.008429 / 0.000200 (0.008229) | 0.000101 / 0.000054 (0.000047) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029182 / 0.037411 (-0.008229) | 0.126284 / 0.014526 (0.111758) | 0.140381 / 0.176557 (-0.036175) | 0.175878 / 0.737135 (-0.561257) | 0.138824 / 0.296338 (-0.157514) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.643658 / 0.215209 (0.428449) | 6.396224 / 2.077655 (4.318569) | 2.600702 / 1.504120 (1.096582) | 2.176721 / 1.541195 (0.635526) | 2.216116 / 1.468490 (0.747626) | 1.235069 / 4.584777 (-3.349708) | 5.457228 / 3.745712 (1.711516) | 3.060455 / 5.269862 (-2.209407) | 2.028123 / 4.565676 (-2.537554) | 0.141617 / 0.424275 (-0.282658) | 0.016596 / 0.007607 (0.008989) | 0.804915 / 0.226044 (0.578870) | 7.968821 / 2.268929 (5.699893) | 3.340650 / 55.444624 (-52.103974) | 2.533620 / 6.876477 (-4.342856) | 2.457388 / 2.142072 (0.315315) | 1.486527 / 4.805227 (-3.318700) | 0.253767 / 6.500664 (-6.246897) | 0.082192 / 0.075469 (0.006723) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.470896 / 1.841788 (-0.370892) | 17.566637 / 8.074308 (9.492329) | 23.144148 / 10.191392 (12.952756) | 0.235510 / 0.680424 (-0.444913) | 0.046051 / 0.534201 (-0.488150) | 0.559954 / 0.579283 (-0.019329) | 0.645390 / 0.434364 (0.211026) | 0.690983 / 0.540337 (0.150646) | 0.776252 / 1.386936 (-0.610684) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.010564 / 0.011353 (-0.000789) | 0.006150 / 0.011008 (-0.004858) | 0.100030 / 0.038508 (0.061522) | 0.036873 / 0.023109 (0.013764) | 0.448508 / 0.275898 (0.172610) | 0.492593 / 0.323480 (0.169113) | 0.007337 / 0.007986 (-0.000648) | 0.004804 / 0.004328 (0.000475) | 0.099218 / 0.004250 (0.094967) | 0.055513 / 0.037052 (0.018461) | 0.462147 / 0.258489 (0.203658) | 0.510229 / 0.293841 (0.216388) | 0.055307 / 0.128546 (-0.073239) | 0.021989 / 0.075646 (-0.053657) | 0.118487 / 0.419271 (-0.300785) | 0.071752 / 0.043533 (0.028219) | 0.456572 / 0.255139 (0.201433) | 0.475160 / 0.283200 (0.191961) | 0.117472 / 0.141683 (-0.024211) | 1.813212 / 1.452155 (0.361058) | 1.908413 / 1.492716 (0.415696) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.352929 / 0.018006 (0.334923) | 0.543874 / 0.000490 (0.543384) | 0.078529 / 0.000200 (0.078329) | 0.000669 / 0.000054 (0.000614) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033157 / 0.037411 (-0.004254) | 0.162503 / 0.014526 (0.147977) | 0.146424 / 0.176557 (-0.030132) | 0.201781 / 0.737135 (-0.535354) | 0.168110 / 0.296338 (-0.128229) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.644205 / 0.215209 (0.428996) | 6.327519 / 2.077655 (4.249865) | 2.728102 / 1.504120 (1.223982) | 2.306426 / 1.541195 (0.765232) | 2.373125 / 1.468490 (0.904635) | 1.350649 / 4.584777 (-3.234128) | 5.652714 / 3.745712 (1.907002) | 3.175335 / 5.269862 (-2.094526) | 2.222902 / 4.565676 (-2.342775) | 0.160609 / 0.424275 (-0.263666) | 0.015596 / 0.007607 (0.007989) | 0.790357 / 0.226044 (0.564313) | 8.289758 / 2.268929 (6.020830) | 3.479215 / 55.444624 (-51.965410) | 2.860063 / 6.876477 (-4.016413) | 2.806720 / 2.142072 (0.664648) | 1.639046 / 4.805227 (-3.166181) | 0.267017 / 6.500664 (-6.233648) | 0.083990 / 0.075469 (0.008521) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.632262 / 1.841788 (-0.209525) | 17.794357 / 8.074308 (9.720049) | 21.203547 / 10.191392 (11.012155) | 0.250899 / 0.680424 (-0.429525) | 0.024502 / 0.534201 (-0.509699) | 0.519960 / 0.579283 (-0.059323) | 0.615412 / 0.434364 (0.181048) | 0.641914 / 0.540337 (0.101577) | 0.772355 / 1.386936 (-0.614581) |

</details>
</details>

![](https://cml.dev/watermark.png#32cc4d10243b0feb69650f007d010971fd861dc1 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009501 / 0.011353 (-0.001852) | 0.005262 / 0.011008 (-0.005747) | 0.100809 / 0.038508 (0.062301) | 0.036601 / 0.023109 (0.013492) | 0.299612 / 0.275898 (0.023714) | 0.366970 / 0.323480 (0.043490) | 0.007879 / 0.007986 (-0.000107) | 0.004216 / 0.004328 (-0.000113) | 0.076749 / 0.004250 (0.072498) | 0.042081 / 0.037052 (0.005029) | 0.299572 / 0.258489 (0.041083) | 0.339687 / 0.293841 (0.045846) | 0.038706 / 0.128546 (-0.089840) | 0.012295 / 0.075646 (-0.063352) | 0.336172 / 0.419271 (-0.083100) | 0.047524 / 0.043533 (0.003992) | 0.296800 / 0.255139 (0.041661) | 0.331592 / 0.283200 (0.048393) | 0.101191 / 0.141683 (-0.040491) | 1.486200 / 1.452155 (0.034046) | 1.509955 / 1.492716 (0.017239) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.204735 / 0.018006 (0.186728) | 0.446381 / 0.000490 (0.445891) | 0.005177 / 0.000200 (0.004977) | 0.000099 / 0.000054 (0.000045) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028655 / 0.037411 (-0.008756) | 0.116559 / 0.014526 (0.102033) | 0.122551 / 0.176557 (-0.054006) | 0.189764 / 0.737135 (-0.547372) | 0.126446 / 0.296338 (-0.169892) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.400104 / 0.215209 (0.184895) | 4.001524 / 2.077655 (1.923869) | 1.779267 / 1.504120 (0.275147) | 1.580168 / 1.541195 (0.038974) | 1.684100 / 1.468490 (0.215610) | 0.703354 / 4.584777 (-3.881423) | 3.828131 / 3.745712 (0.082419) | 2.098500 / 5.269862 (-3.171362) | 1.331161 / 4.565676 (-3.234516) | 0.085417 / 0.424275 (-0.338858) | 0.012380 / 0.007607 (0.004772) | 0.504189 / 0.226044 (0.278144) | 5.094672 / 2.268929 (2.825743) | 2.264352 / 55.444624 (-53.180272) | 1.909573 / 6.876477 (-4.966904) | 2.005425 / 2.142072 (-0.136648) | 0.840893 / 4.805227 (-3.964335) | 0.164689 / 6.500664 (-6.335975) | 0.062754 / 0.075469 (-0.012715) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.250001 / 1.841788 (-0.591786) | 14.993313 / 8.074308 (6.919005) | 14.880601 / 10.191392 (4.689209) | 0.175141 / 0.680424 (-0.505283) | 0.028952 / 0.534201 (-0.505249) | 0.447073 / 0.579283 (-0.132210) | 0.445993 / 0.434364 (0.011629) | 0.525527 / 0.540337 (-0.014811) | 0.613156 / 1.386936 (-0.773780) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007796 / 0.011353 (-0.003557) | 0.005399 / 0.011008 (-0.005609) | 0.078240 / 0.038508 (0.039732) | 0.035303 / 0.023109 (0.012193) | 0.364603 / 0.275898 (0.088705) | 0.400794 / 0.323480 (0.077314) | 0.006152 / 0.007986 (-0.001834) | 0.004324 / 0.004328 (-0.000004) | 0.074949 / 0.004250 (0.070698) | 0.051939 / 0.037052 (0.014887) | 0.377079 / 0.258489 (0.118590) | 0.413630 / 0.293841 (0.119789) | 0.037567 / 0.128546 (-0.090979) | 0.012793 / 0.075646 (-0.062854) | 0.089013 / 0.419271 (-0.330258) | 0.050748 / 0.043533 (0.007215) | 0.370100 / 0.255139 (0.114961) | 0.384838 / 0.283200 (0.101638) | 0.105840 / 0.141683 (-0.035843) | 1.476490 / 1.452155 (0.024335) | 1.544688 / 1.492716 (0.051972) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.220987 / 0.018006 (0.202981) | 0.443801 / 0.000490 (0.443311) | 0.005747 / 0.000200 (0.005547) | 0.000106 / 0.000054 (0.000051) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030187 / 0.037411 (-0.007225) | 0.118230 / 0.014526 (0.103704) | 0.126810 / 0.176557 (-0.049746) | 0.200482 / 0.737135 (-0.536654) | 0.130831 / 0.296338 (-0.165507) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.423231 / 0.215209 (0.208022) | 4.196576 / 2.077655 (2.118921) | 1.992919 / 1.504120 (0.488799) | 1.809172 / 1.541195 (0.267977) | 1.932706 / 1.468490 (0.464216) | 0.727319 / 4.584777 (-3.857458) | 3.833295 / 3.745712 (0.087583) | 3.527005 / 5.269862 (-1.742857) | 1.937348 / 4.565676 (-2.628329) | 0.088713 / 0.424275 (-0.335562) | 0.012711 / 0.007607 (0.005104) | 0.531385 / 0.226044 (0.305341) | 5.308051 / 2.268929 (3.039123) | 2.493494 / 55.444624 (-52.951131) | 2.168359 / 6.876477 (-4.708118) | 2.258160 / 2.142072 (0.116088) | 0.865629 / 4.805227 (-3.939598) | 0.171281 / 6.500664 (-6.329383) | 0.065746 / 0.075469 (-0.009723) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.290378 / 1.841788 (-0.551409) | 15.900804 / 8.074308 (7.826496) | 14.809614 / 10.191392 (4.618222) | 0.177287 / 0.680424 (-0.503137) | 0.017875 / 0.534201 (-0.516326) | 0.429646 / 0.579283 (-0.149637) | 0.451646 / 0.434364 (0.017282) | 0.545669 / 0.540337 (0.005332) | 0.633215 / 1.386936 (-0.753721) |

</details>
</details>

![](https://cml.dev/watermark.png#2c67b5f4bc9cea088e977a135644d38da8c144ff "CML watermark")