Use `filelock` package for file locking
State: closed | Author: mariosasko (COLLABORATOR)

Use the `filelock` package instead of `datasets.utils.filelock` for file locking to be consistent with `huggingface_hub` and not to be responsible for improving the `filelock` capabilities 🙂. 

(Reverts https://github.com/huggingface/datasets/pull/859, but these `INFO` logs are not printed by default (anymore?), so this should be okay)

 

Comments: <details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005431 / 0.011353 (-0.005922) | 0.003255 / 0.011008 (-0.007753) | 0.062867 / 0.038508 (0.024359) | 0.051917 / 0.023109 (0.028808) | 0.254229 / 0.275898 (-0.021669) | 0.276949 / 0.323480 (-0.046531) | 0.002868 / 0.007986 (-0.005117) | 0.002539 / 0.004328 (-0.001789) | 0.048366 / 0.004250 (0.044115) | 0.038497 / 0.037052 (0.001445) | 0.252158 / 0.258489 (-0.006332) | 0.288868 / 0.293841 (-0.004973) | 0.027956 / 0.128546 (-0.100591) | 0.010500 / 0.075646 (-0.065147) | 0.209263 / 0.419271 (-0.210008) | 0.035415 / 0.043533 (-0.008118) | 0.253104 / 0.255139 (-0.002035) | 0.274646 / 0.283200 (-0.008554) | 0.019923 / 0.141683 (-0.121760) | 1.081870 / 1.452155 (-0.370285) | 1.157159 / 1.492716 (-0.335557) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.097420 / 0.018006 (0.079414) | 0.315021 / 0.000490 (0.314531) | 0.000218 / 0.000200 (0.000018) | 0.000049 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018826 / 0.037411 (-0.018585) | 0.061921 / 0.014526 (0.047395) | 0.086825 / 0.176557 (-0.089731) | 0.120606 / 0.737135 (-0.616529) | 0.074344 / 0.296338 (-0.221994) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.283238 / 0.215209 (0.068028) | 2.771817 / 2.077655 (0.694162) | 1.500194 / 1.504120 (-0.003926) | 1.379286 / 1.541195 (-0.161908) | 1.447747 / 1.468490 (-0.020743) | 0.587176 / 4.584777 (-3.997601) | 2.411260 / 3.745712 (-1.334452) | 2.897682 / 5.269862 (-2.372180) | 1.821720 / 4.565676 (-2.743957) | 0.063299 / 0.424275 (-0.360976) | 0.004969 / 0.007607 (-0.002639) | 0.346417 / 0.226044 (0.120373) | 3.432936 / 2.268929 (1.164007) | 1.898662 / 55.444624 (-53.545963) | 1.624339 / 6.876477 (-5.252138) | 1.641653 / 2.142072 (-0.500419) | 0.655773 / 4.805227 (-4.149454) | 0.118588 / 6.500664 (-6.382076) | 0.043919 / 0.075469 (-0.031551) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.949466 / 1.841788 (-0.892322) | 12.378025 / 8.074308 (4.303717) | 10.750942 / 10.191392 (0.559550) | 0.146575 / 0.680424 (-0.533849) | 0.015453 / 0.534201 (-0.518748) | 0.290608 / 0.579283 (-0.288676) | 0.273000 / 0.434364 (-0.161364) | 0.328019 / 0.540337 (-0.212318) | 0.417396 / 1.386936 (-0.969540) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005363 / 0.011353 (-0.005990) | 0.003421 / 0.011008 (-0.007587) | 0.049429 / 0.038508 (0.010920) | 0.052774 / 0.023109 (0.029664) | 0.274058 / 0.275898 (-0.001840) | 0.297307 / 0.323480 (-0.026173) | 0.004000 / 0.007986 (-0.003986) | 0.002463 / 0.004328 (-0.001866) | 0.048824 / 0.004250 (0.044574) | 0.041064 / 0.037052 (0.004012) | 0.279066 / 0.258489 (0.020577) | 0.302420 / 0.293841 (0.008579) | 0.029665 / 0.128546 (-0.098881) | 0.010628 / 0.075646 (-0.065018) | 0.057678 / 0.419271 (-0.361594) | 0.032731 / 0.043533 (-0.010802) | 0.274662 / 0.255139 (0.019523) | 0.291878 / 0.283200 (0.008678) | 0.018820 / 0.141683 (-0.122863) | 1.124042 / 1.452155 (-0.328112) | 1.175020 / 1.492716 (-0.317697) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.099419 / 0.018006 (0.081413) | 0.311511 / 0.000490 (0.311022) | 0.000228 / 0.000200 (0.000028) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022478 / 0.037411 (-0.014933) | 0.071955 / 0.014526 (0.057429) | 0.081423 / 0.176557 (-0.095134) | 0.119574 / 0.737135 (-0.617561) | 0.084724 / 0.296338 (-0.211615) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.295537 / 0.215209 (0.080328) | 2.893855 / 2.077655 (0.816201) | 1.602065 / 1.504120 (0.097945) | 1.478193 / 1.541195 (-0.063002) | 1.508250 / 1.468490 (0.039760) | 0.566140 / 4.584777 (-4.018637) | 2.455474 / 3.745712 (-1.290238) | 2.849525 / 5.269862 (-2.420337) | 1.763830 / 4.565676 (-2.801846) | 0.062375 / 0.424275 (-0.361900) | 0.004992 / 0.007607 (-0.002615) | 0.346068 / 0.226044 (0.120023) | 3.452421 / 2.268929 (1.183492) | 1.970346 / 55.444624 (-53.474278) | 1.690865 / 6.876477 (-5.185612) | 1.705358 / 2.142072 (-0.436714) | 0.644261 / 4.805227 (-4.160967) | 0.120596 / 6.500664 (-6.380068) | 0.042699 / 0.075469 (-0.032770) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.980506 / 1.841788 (-0.861281) | 12.401901 / 8.074308 (4.327593) | 11.169413 / 10.191392 (0.978021) | 0.142540 / 0.680424 (-0.537884) | 0.015730 / 0.534201 (-0.518471) | 0.288871 / 0.579283 (-0.290412) | 0.287487 / 0.434364 (-0.146877) | 0.325133 / 0.540337 (-0.215204) | 0.417979 / 1.386936 (-0.968957) |

</details>
</details>

![](https://cml.dev/watermark.png#965685891db0d06979490aaebab72d5dc628e42b "CML watermark")


_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005062 / 0.011353 (-0.006291) | 0.003024 / 0.011008 (-0.007984) | 0.061801 / 0.038508 (0.023293) | 0.048934 / 0.023109 (0.025825) | 0.248024 / 0.275898 (-0.027874) | 0.265665 / 0.323480 (-0.057815) | 0.003885 / 0.007986 (-0.004100) | 0.002371 / 0.004328 (-0.001957) | 0.047895 / 0.004250 (0.043644) | 0.039015 / 0.037052 (0.001963) | 0.252320 / 0.258489 (-0.006169) | 0.286533 / 0.293841 (-0.007308) | 0.027694 / 0.128546 (-0.100852) | 0.010254 / 0.075646 (-0.065392) | 0.206586 / 0.419271 (-0.212685) | 0.035681 / 0.043533 (-0.007852) | 0.251645 / 0.255139 (-0.003494) | 0.285462 / 0.283200 (0.002262) | 0.017326 / 0.141683 (-0.124357) | 1.086927 / 1.452155 (-0.365228) | 1.153172 / 1.492716 (-0.339545) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093020 / 0.018006 (0.075014) | 0.300018 / 0.000490 (0.299528) | 0.000208 / 0.000200 (0.000008) | 0.000047 / 0.000054 (-0.000008) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018828 / 0.037411 (-0.018584) | 0.062569 / 0.014526 (0.048043) | 0.074130 / 0.176557 (-0.102427) | 0.119304 / 0.737135 (-0.617832) | 0.076409 / 0.296338 (-0.219930) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.285938 / 0.215209 (0.070729) | 2.780662 / 2.077655 (0.703007) | 1.522401 / 1.504120 (0.018281) | 1.392475 / 1.541195 (-0.148720) | 1.412517 / 1.468490 (-0.055973) | 0.562768 / 4.584777 (-4.022009) | 2.421406 / 3.745712 (-1.324306) | 2.786271 / 5.269862 (-2.483591) | 1.737193 / 4.565676 (-2.828484) | 0.062775 / 0.424275 (-0.361500) | 0.004908 / 0.007607 (-0.002699) | 0.345070 / 0.226044 (0.119026) | 3.383700 / 2.268929 (1.114771) | 1.795974 / 55.444624 (-53.648651) | 1.527656 / 6.876477 (-5.348820) | 1.514035 / 2.142072 (-0.628037) | 0.647652 / 4.805227 (-4.157575) | 0.120121 / 6.500664 (-6.380543) | 0.042259 / 0.075469 (-0.033210) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.948951 / 1.841788 (-0.892837) | 11.514971 / 8.074308 (3.440663) | 10.722668 / 10.191392 (0.531276) | 0.143034 / 0.680424 (-0.537390) | 0.014800 / 0.534201 (-0.519401) | 0.286189 / 0.579283 (-0.293094) | 0.270735 / 0.434364 (-0.163629) | 0.323907 / 0.540337 (-0.216430) | 0.417569 / 1.386936 (-0.969367) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005670 / 0.011353 (-0.005683) | 0.003238 / 0.011008 (-0.007770) | 0.048520 / 0.038508 (0.010012) | 0.051341 / 0.023109 (0.028232) | 0.273883 / 0.275898 (-0.002015) | 0.295165 / 0.323480 (-0.028315) | 0.004755 / 0.007986 (-0.003231) | 0.002471 / 0.004328 (-0.001857) | 0.047487 / 0.004250 (0.043237) | 0.040225 / 0.037052 (0.003172) | 0.276758 / 0.258489 (0.018269) | 0.301182 / 0.293841 (0.007341) | 0.029749 / 0.128546 (-0.098797) | 0.010340 / 0.075646 (-0.065306) | 0.057193 / 0.419271 (-0.362079) | 0.033067 / 0.043533 (-0.010466) | 0.272716 / 0.255139 (0.017577) | 0.292301 / 0.283200 (0.009101) | 0.019075 / 0.141683 (-0.122608) | 1.101778 / 1.452155 (-0.350376) | 1.173573 / 1.492716 (-0.319143) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091008 / 0.018006 (0.073002) | 0.300749 / 0.000490 (0.300259) | 0.000218 / 0.000200 (0.000018) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021760 / 0.037411 (-0.015651) | 0.071407 / 0.014526 (0.056881) | 0.081151 / 0.176557 (-0.095406) | 0.120140 / 0.737135 (-0.616995) | 0.082408 / 0.296338 (-0.213931) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.294828 / 0.215209 (0.079619) | 2.880701 / 2.077655 (0.803047) | 1.604187 / 1.504120 (0.100068) | 1.479236 / 1.541195 (-0.061959) | 1.498875 / 1.468490 (0.030385) | 0.561950 / 4.584777 (-4.022827) | 2.462531 / 3.745712 (-1.283181) | 2.800905 / 5.269862 (-2.468957) | 1.746535 / 4.565676 (-2.819141) | 0.062732 / 0.424275 (-0.361544) | 0.004932 / 0.007607 (-0.002675) | 0.347125 / 0.226044 (0.121081) | 3.431343 / 2.268929 (1.162415) | 1.964999 / 55.444624 (-53.479625) | 1.669709 / 6.876477 (-5.206768) | 1.675148 / 2.142072 (-0.466924) | 0.635436 / 4.805227 (-4.169792) | 0.116598 / 6.500664 (-6.384066) | 0.041447 / 0.075469 (-0.034022) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.975751 / 1.841788 (-0.866037) | 12.060246 / 8.074308 (3.985938) | 10.871641 / 10.191392 (0.680249) | 0.142936 / 0.680424 (-0.537488) | 0.015779 / 0.534201 (-0.518422) | 0.287120 / 0.579283 (-0.292163) | 0.283963 / 0.434364 (-0.150401) | 0.341231 / 0.540337 (-0.199107) | 0.419518 / 1.386936 (-0.967418) |

</details>
</details>

![](https://cml.dev/watermark.png#0943ff0072dcef473530d8a494f314048f3a3d51 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005105 / 0.011353 (-0.006248) | 0.002855 / 0.011008 (-0.008153) | 0.062044 / 0.038508 (0.023536) | 0.052948 / 0.023109 (0.029839) | 0.249841 / 0.275898 (-0.026057) | 0.276687 / 0.323480 (-0.046792) | 0.003792 / 0.007986 (-0.004194) | 0.002385 / 0.004328 (-0.001943) | 0.048648 / 0.004250 (0.044398) | 0.038317 / 0.037052 (0.001264) | 0.255235 / 0.258489 (-0.003254) | 0.287870 / 0.293841 (-0.005971) | 0.027429 / 0.128546 (-0.101117) | 0.010182 / 0.075646 (-0.065464) | 0.206980 / 0.419271 (-0.212291) | 0.035444 / 0.043533 (-0.008089) | 0.255073 / 0.255139 (-0.000066) | 0.270636 / 0.283200 (-0.012563) | 0.018003 / 0.141683 (-0.123680) | 1.124691 / 1.452155 (-0.327463) | 1.191872 / 1.492716 (-0.300844) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.088824 / 0.018006 (0.070818) | 0.302771 / 0.000490 (0.302281) | 0.000210 / 0.000200 (0.000010) | 0.000048 / 0.000054 (-0.000006) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018102 / 0.037411 (-0.019310) | 0.062131 / 0.014526 (0.047605) | 0.073230 / 0.176557 (-0.103327) | 0.119789 / 0.737135 (-0.617346) | 0.074804 / 0.296338 (-0.221534) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.293244 / 0.215209 (0.078035) | 2.891401 / 2.077655 (0.813746) | 1.504481 / 1.504120 (0.000361) | 1.381251 / 1.541195 (-0.159944) | 1.387245 / 1.468490 (-0.081245) | 0.552732 / 4.584777 (-4.032045) | 2.386439 / 3.745712 (-1.359273) | 2.718918 / 5.269862 (-2.550944) | 1.725401 / 4.565676 (-2.840275) | 0.061946 / 0.424275 (-0.362329) | 0.004957 / 0.007607 (-0.002650) | 0.342776 / 0.226044 (0.116731) | 3.418911 / 2.268929 (1.149983) | 1.838283 / 55.444624 (-53.606341) | 1.538013 / 6.876477 (-5.338464) | 1.545144 / 2.142072 (-0.596928) | 0.637857 / 4.805227 (-4.167370) | 0.116451 / 6.500664 (-6.384213) | 0.042228 / 0.075469 (-0.033241) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.943575 / 1.841788 (-0.898212) | 11.492939 / 8.074308 (3.418631) | 10.601605 / 10.191392 (0.410212) | 0.139084 / 0.680424 (-0.541340) | 0.013691 / 0.534201 (-0.520510) | 0.286696 / 0.579283 (-0.292587) | 0.259979 / 0.434364 (-0.174385) | 0.322578 / 0.540337 (-0.217759) | 0.411950 / 1.386936 (-0.974986) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005168 / 0.011353 (-0.006185) | 0.003238 / 0.011008 (-0.007770) | 0.049028 / 0.038508 (0.010520) | 0.052930 / 0.023109 (0.029821) | 0.274750 / 0.275898 (-0.001148) | 0.294023 / 0.323480 (-0.029457) | 0.003829 / 0.007986 (-0.004157) | 0.002372 / 0.004328 (-0.001956) | 0.048689 / 0.004250 (0.044439) | 0.040056 / 0.037052 (0.003003) | 0.280147 / 0.258489 (0.021658) | 0.304871 / 0.293841 (0.011030) | 0.028734 / 0.128546 (-0.099812) | 0.010624 / 0.075646 (-0.065022) | 0.058705 / 0.419271 (-0.360566) | 0.032140 / 0.043533 (-0.011393) | 0.276702 / 0.255139 (0.021563) | 0.293186 / 0.283200 (0.009987) | 0.018124 / 0.141683 (-0.123559) | 1.139398 / 1.452155 (-0.312757) | 1.174862 / 1.492716 (-0.317855) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.087627 / 0.018006 (0.069620) | 0.298376 / 0.000490 (0.297886) | 0.000238 / 0.000200 (0.000038) | 0.000052 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021344 / 0.037411 (-0.016067) | 0.070208 / 0.014526 (0.055682) | 0.081177 / 0.176557 (-0.095380) | 0.120170 / 0.737135 (-0.616965) | 0.082472 / 0.296338 (-0.213866) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.293227 / 0.215209 (0.078018) | 2.844619 / 2.077655 (0.766964) | 1.586922 / 1.504120 (0.082803) | 1.460256 / 1.541195 (-0.080938) | 1.475955 / 1.468490 (0.007465) | 0.553226 / 4.584777 (-4.031551) | 2.418869 / 3.745712 (-1.326843) | 2.709256 / 5.269862 (-2.560606) | 1.705935 / 4.565676 (-2.859741) | 0.062391 / 0.424275 (-0.361884) | 0.004929 / 0.007607 (-0.002678) | 0.350358 / 0.226044 (0.124313) | 3.448824 / 2.268929 (1.179896) | 1.929451 / 55.444624 (-53.515174) | 1.669438 / 6.876477 (-5.207038) | 1.660923 / 2.142072 (-0.481150) | 0.633107 / 4.805227 (-4.172120) | 0.114657 / 6.500664 (-6.386007) | 0.041256 / 0.075469 (-0.034214) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.968408 / 1.841788 (-0.873380) | 11.749754 / 8.074308 (3.675446) | 10.796670 / 10.191392 (0.605278) | 0.128881 / 0.680424 (-0.551543) | 0.015326 / 0.534201 (-0.518875) | 0.286407 / 0.579283 (-0.292876) | 0.276324 / 0.434364 (-0.158040) | 0.326201 / 0.540337 (-0.214136) | 0.419854 / 1.386936 (-0.967082) |

</details>
</details>

![](https://cml.dev/watermark.png#1731d5a8cd103533ef6b438b4429ab51d3a6a0ce "CML watermark")