Jax requires jaxlib
State: closed | Author: lhoestq (MEMBER)

close https://github.com/huggingface/datasets/issues/5666

Comments: _The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008592 / 0.011353 (-0.002761) | 0.005182 / 0.011008 (-0.005826) | 0.097916 / 0.038508 (0.059408) | 0.034612 / 0.023109 (0.011503) | 0.313760 / 0.275898 (0.037862) | 0.353422 / 0.323480 (0.029942) | 0.005880 / 0.007986 (-0.002106) | 0.004123 / 0.004328 (-0.000205) | 0.073634 / 0.004250 (0.069384) | 0.049349 / 0.037052 (0.012297) | 0.317381 / 0.258489 (0.058892) | 0.365821 / 0.293841 (0.071980) | 0.036482 / 0.128546 (-0.092065) | 0.012126 / 0.075646 (-0.063521) | 0.334640 / 0.419271 (-0.084631) | 0.050551 / 0.043533 (0.007018) | 0.310472 / 0.255139 (0.055333) | 0.349049 / 0.283200 (0.065850) | 0.101343 / 0.141683 (-0.040340) | 1.447903 / 1.452155 (-0.004252) | 1.518793 / 1.492716 (0.026077) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.210971 / 0.018006 (0.192965) | 0.449471 / 0.000490 (0.448982) | 0.003596 / 0.000200 (0.003396) | 0.000084 / 0.000054 (0.000029) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027386 / 0.037411 (-0.010025) | 0.112683 / 0.014526 (0.098157) | 0.117603 / 0.176557 (-0.058954) | 0.174186 / 0.737135 (-0.562949) | 0.123510 / 0.296338 (-0.172829) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.422595 / 0.215209 (0.207386) | 4.224713 / 2.077655 (2.147058) | 2.006359 / 1.504120 (0.502240) | 1.823767 / 1.541195 (0.282572) | 1.898340 / 1.468490 (0.429849) | 0.721656 / 4.584777 (-3.863121) | 3.823498 / 3.745712 (0.077785) | 2.172380 / 5.269862 (-3.097481) | 1.469773 / 4.565676 (-3.095904) | 0.086978 / 0.424275 (-0.337297) | 0.012642 / 0.007607 (0.005035) | 0.517830 / 0.226044 (0.291785) | 5.171150 / 2.268929 (2.902221) | 2.495238 / 55.444624 (-52.949386) | 2.114380 / 6.876477 (-4.762097) | 2.274329 / 2.142072 (0.132257) | 0.863855 / 4.805227 (-3.941372) | 0.174127 / 6.500664 (-6.326537) | 0.065939 / 0.075469 (-0.009530) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.208831 / 1.841788 (-0.632957) | 15.016704 / 8.074308 (6.942396) | 14.721231 / 10.191392 (4.529839) | 0.144140 / 0.680424 (-0.536284) | 0.017781 / 0.534201 (-0.516420) | 0.425679 / 0.579283 (-0.153604) | 0.416747 / 0.434364 (-0.017617) | 0.490160 / 0.540337 (-0.050177) | 0.583639 / 1.386936 (-0.803297) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007670 / 0.011353 (-0.003683) | 0.005383 / 0.011008 (-0.005626) | 0.075756 / 0.038508 (0.037248) | 0.033373 / 0.023109 (0.010263) | 0.341017 / 0.275898 (0.065119) | 0.378890 / 0.323480 (0.055410) | 0.005945 / 0.007986 (-0.002040) | 0.004179 / 0.004328 (-0.000150) | 0.074588 / 0.004250 (0.070337) | 0.048564 / 0.037052 (0.011511) | 0.338774 / 0.258489 (0.080285) | 0.391081 / 0.293841 (0.097240) | 0.036659 / 0.128546 (-0.091887) | 0.012241 / 0.075646 (-0.063406) | 0.086910 / 0.419271 (-0.332361) | 0.049745 / 0.043533 (0.006212) | 0.332810 / 0.255139 (0.077671) | 0.360317 / 0.283200 (0.077117) | 0.103399 / 0.141683 (-0.038283) | 1.456754 / 1.452155 (0.004599) | 1.542644 / 1.492716 (0.049928) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.207182 / 0.018006 (0.189176) | 0.455659 / 0.000490 (0.455169) | 0.003609 / 0.000200 (0.003409) | 0.000092 / 0.000054 (0.000038) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029556 / 0.037411 (-0.007856) | 0.114215 / 0.014526 (0.099690) | 0.127721 / 0.176557 (-0.048836) | 0.177070 / 0.737135 (-0.560065) | 0.128840 / 0.296338 (-0.167499) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.428176 / 0.215209 (0.212967) | 4.274324 / 2.077655 (2.196669) | 2.020058 / 1.504120 (0.515938) | 1.823343 / 1.541195 (0.282148) | 1.924688 / 1.468490 (0.456198) | 0.719195 / 4.584777 (-3.865582) | 3.760445 / 3.745712 (0.014733) | 2.133813 / 5.269862 (-3.136049) | 1.364876 / 4.565676 (-3.200801) | 0.087523 / 0.424275 (-0.336752) | 0.013712 / 0.007607 (0.006105) | 0.528403 / 0.226044 (0.302359) | 5.307780 / 2.268929 (3.038851) | 2.496747 / 55.444624 (-52.947877) | 2.169136 / 6.876477 (-4.707341) | 2.235719 / 2.142072 (0.093646) | 0.875281 / 4.805227 (-3.929946) | 0.172369 / 6.500664 (-6.328295) | 0.064667 / 0.075469 (-0.010802) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.262594 / 1.841788 (-0.579193) | 15.182681 / 8.074308 (7.108373) | 14.725663 / 10.191392 (4.534271) | 0.180961 / 0.680424 (-0.499462) | 0.017632 / 0.534201 (-0.516569) | 0.427531 / 0.579283 (-0.151752) | 0.431741 / 0.434364 (-0.002622) | 0.503251 / 0.540337 (-0.037087) | 0.597423 / 1.386936 (-0.789513) |

</details>
</details>

![](https://cml.dev/watermark.png#f4cf224dcb1043a272971ed331a214cf65c504be "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009761 / 0.011353 (-0.001592) | 0.006779 / 0.011008 (-0.004229) | 0.132786 / 0.038508 (0.094277) | 0.037721 / 0.023109 (0.014611) | 0.435685 / 0.275898 (0.159787) | 0.447488 / 0.323480 (0.124009) | 0.006848 / 0.007986 (-0.001137) | 0.005099 / 0.004328 (0.000771) | 0.097384 / 0.004250 (0.093133) | 0.056663 / 0.037052 (0.019610) | 0.463407 / 0.258489 (0.204918) | 0.502544 / 0.293841 (0.208703) | 0.053817 / 0.128546 (-0.074729) | 0.020253 / 0.075646 (-0.055393) | 0.446653 / 0.419271 (0.027382) | 0.064465 / 0.043533 (0.020932) | 0.455375 / 0.255139 (0.200236) | 0.458378 / 0.283200 (0.175178) | 0.109124 / 0.141683 (-0.032559) | 1.957338 / 1.452155 (0.505184) | 1.960391 / 1.492716 (0.467674) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.219566 / 0.018006 (0.201560) | 0.558181 / 0.000490 (0.557691) | 0.004678 / 0.000200 (0.004478) | 0.000125 / 0.000054 (0.000071) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032643 / 0.037411 (-0.004768) | 0.147375 / 0.014526 (0.132849) | 0.130821 / 0.176557 (-0.045736) | 0.203202 / 0.737135 (-0.533933) | 0.145186 / 0.296338 (-0.151153) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.665773 / 0.215209 (0.450564) | 6.674021 / 2.077655 (4.596366) | 2.662372 / 1.504120 (1.158253) | 2.333327 / 1.541195 (0.792132) | 2.221413 / 1.468490 (0.752923) | 1.287001 / 4.584777 (-3.297776) | 5.534326 / 3.745712 (1.788614) | 3.188809 / 5.269862 (-2.081052) | 2.261717 / 4.565676 (-2.303960) | 0.151910 / 0.424275 (-0.272366) | 0.020509 / 0.007607 (0.012902) | 0.863608 / 0.226044 (0.637564) | 8.442155 / 2.268929 (6.173227) | 3.438260 / 55.444624 (-52.006364) | 2.692503 / 6.876477 (-4.183974) | 2.810997 / 2.142072 (0.668925) | 1.477345 / 4.805227 (-3.327882) | 0.261942 / 6.500664 (-6.238722) | 0.086347 / 0.075469 (0.010878) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.529072 / 1.841788 (-0.312716) | 17.213019 / 8.074308 (9.138711) | 21.887309 / 10.191392 (11.695917) | 0.259660 / 0.680424 (-0.420763) | 0.027916 / 0.534201 (-0.506285) | 0.554103 / 0.579283 (-0.025180) | 0.614566 / 0.434364 (0.180202) | 0.700456 / 0.540337 (0.160119) | 0.756860 / 1.386936 (-0.630077) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009267 / 0.011353 (-0.002086) | 0.006414 / 0.011008 (-0.004594) | 0.102404 / 0.038508 (0.063896) | 0.034885 / 0.023109 (0.011776) | 0.413191 / 0.275898 (0.137293) | 0.483901 / 0.323480 (0.160422) | 0.006614 / 0.007986 (-0.001372) | 0.004608 / 0.004328 (0.000280) | 0.096717 / 0.004250 (0.092467) | 0.055123 / 0.037052 (0.018071) | 0.417786 / 0.258489 (0.159297) | 0.490886 / 0.293841 (0.197045) | 0.056951 / 0.128546 (-0.071595) | 0.021073 / 0.075646 (-0.054574) | 0.116576 / 0.419271 (-0.302695) | 0.063968 / 0.043533 (0.020435) | 0.420495 / 0.255139 (0.165356) | 0.449667 / 0.283200 (0.166467) | 0.115318 / 0.141683 (-0.026365) | 1.899398 / 1.452155 (0.447243) | 1.992175 / 1.492716 (0.499459) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.233076 / 0.018006 (0.215070) | 0.518377 / 0.000490 (0.517887) | 0.000809 / 0.000200 (0.000609) | 0.000101 / 0.000054 (0.000047) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030951 / 0.037411 (-0.006460) | 0.134940 / 0.014526 (0.120414) | 0.147789 / 0.176557 (-0.028767) | 0.205854 / 0.737135 (-0.531281) | 0.146726 / 0.296338 (-0.149613) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.648006 / 0.215209 (0.432797) | 6.416688 / 2.077655 (4.339033) | 2.696462 / 1.504120 (1.192342) | 2.293071 / 1.541195 (0.751877) | 2.319426 / 1.468490 (0.850935) | 1.332398 / 4.584777 (-3.252379) | 5.706956 / 3.745712 (1.961244) | 4.464473 / 5.269862 (-0.805388) | 2.817364 / 4.565676 (-1.748312) | 0.157595 / 0.424275 (-0.266680) | 0.015721 / 0.007607 (0.008114) | 0.806055 / 0.226044 (0.580010) | 7.927795 / 2.268929 (5.658866) | 3.461251 / 55.444624 (-51.983373) | 2.664466 / 6.876477 (-4.212010) | 2.660041 / 2.142072 (0.517968) | 1.531135 / 4.805227 (-3.274092) | 0.260293 / 6.500664 (-6.240371) | 0.077440 / 0.075469 (0.001971) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.687325 / 1.841788 (-0.154463) | 17.905080 / 8.074308 (9.830772) | 21.046794 / 10.191392 (10.855402) | 0.245335 / 0.680424 (-0.435089) | 0.026830 / 0.534201 (-0.507371) | 0.510798 / 0.579283 (-0.068485) | 0.590041 / 0.434364 (0.155677) | 0.607440 / 0.540337 (0.067102) | 0.725030 / 1.386936 (-0.661906) |

</details>
</details>

![](https://cml.dev/watermark.png#91dcb3636e410a249177f5e0508ed101ad7ee25b "CML watermark")


I self-assigned #5666 and I was working on it... without success: https://github.com/huggingface/datasets/tree/fix-5666

I think your approach is the right one because installation of jax is not trivial...

Next time it would be better that you self-assign an issue before working on it, so that we avoid duplicate work... :sweat_smile: 

Oh sorry I forgot to self assign this time

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008436 / 0.011353 (-0.002917) | 0.005702 / 0.011008 (-0.005306) | 0.113518 / 0.038508 (0.075010) | 0.039639 / 0.023109 (0.016530) | 0.353200 / 0.275898 (0.077302) | 0.382428 / 0.323480 (0.058948) | 0.007419 / 0.007986 (-0.000566) | 0.005640 / 0.004328 (0.001311) | 0.083905 / 0.004250 (0.079655) | 0.053258 / 0.037052 (0.016205) | 0.371069 / 0.258489 (0.112580) | 0.390439 / 0.293841 (0.096598) | 0.042679 / 0.128546 (-0.085867) | 0.013438 / 0.075646 (-0.062208) | 0.390116 / 0.419271 (-0.029155) | 0.068782 / 0.043533 (0.025249) | 0.352620 / 0.255139 (0.097481) | 0.371939 / 0.283200 (0.088739) | 0.126157 / 0.141683 (-0.015525) | 1.694638 / 1.452155 (0.242484) | 1.799211 / 1.492716 (0.306495) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.260099 / 0.018006 (0.242092) | 0.489852 / 0.000490 (0.489362) | 0.012549 / 0.000200 (0.012349) | 0.000275 / 0.000054 (0.000221) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032235 / 0.037411 (-0.005177) | 0.125325 / 0.014526 (0.110799) | 0.137242 / 0.176557 (-0.039315) | 0.206566 / 0.737135 (-0.530570) | 0.143260 / 0.296338 (-0.153078) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.478510 / 0.215209 (0.263301) | 4.746439 / 2.077655 (2.668784) | 2.195072 / 1.504120 (0.690952) | 1.958163 / 1.541195 (0.416969) | 2.028566 / 1.468490 (0.560075) | 0.821289 / 4.584777 (-3.763488) | 4.765529 / 3.745712 (1.019817) | 2.378753 / 5.269862 (-2.891108) | 1.514776 / 4.565676 (-3.050900) | 0.100673 / 0.424275 (-0.323602) | 0.014720 / 0.007607 (0.007113) | 0.606388 / 0.226044 (0.380343) | 5.975285 / 2.268929 (3.706357) | 2.866762 / 55.444624 (-52.577862) | 2.392132 / 6.876477 (-4.484345) | 2.546487 / 2.142072 (0.404415) | 0.982394 / 4.805227 (-3.822833) | 0.201195 / 6.500664 (-6.299469) | 0.077781 / 0.075469 (0.002312) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.420613 / 1.841788 (-0.421174) | 17.743030 / 8.074308 (9.668722) | 16.752344 / 10.191392 (6.560951) | 0.167464 / 0.680424 (-0.512960) | 0.020908 / 0.534201 (-0.513293) | 0.502919 / 0.579283 (-0.076364) | 0.506375 / 0.434364 (0.072011) | 0.602695 / 0.540337 (0.062358) | 0.689398 / 1.386936 (-0.697538) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008713 / 0.011353 (-0.002640) | 0.006152 / 0.011008 (-0.004856) | 0.091264 / 0.038508 (0.052756) | 0.040284 / 0.023109 (0.017174) | 0.417598 / 0.275898 (0.141700) | 0.460141 / 0.323480 (0.136661) | 0.006589 / 0.007986 (-0.001397) | 0.004671 / 0.004328 (0.000343) | 0.089360 / 0.004250 (0.085110) | 0.055113 / 0.037052 (0.018061) | 0.415241 / 0.258489 (0.156752) | 0.470566 / 0.293841 (0.176725) | 0.042963 / 0.128546 (-0.085584) | 0.014421 / 0.075646 (-0.061225) | 0.106333 / 0.419271 (-0.312939) | 0.057810 / 0.043533 (0.014277) | 0.417889 / 0.255139 (0.162750) | 0.444236 / 0.283200 (0.161036) | 0.119508 / 0.141683 (-0.022175) | 1.736209 / 1.452155 (0.284055) | 1.790319 / 1.492716 (0.297602) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.219184 / 0.018006 (0.201178) | 0.493931 / 0.000490 (0.493441) | 0.006727 / 0.000200 (0.006527) | 0.000103 / 0.000054 (0.000049) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034415 / 0.037411 (-0.002996) | 0.132165 / 0.014526 (0.117639) | 0.143138 / 0.176557 (-0.033418) | 0.200052 / 0.737135 (-0.537083) | 0.148906 / 0.296338 (-0.147433) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.483686 / 0.215209 (0.268476) | 4.849874 / 2.077655 (2.772220) | 2.374276 / 1.504120 (0.870156) | 2.168334 / 1.541195 (0.627139) | 2.285983 / 1.468490 (0.817493) | 0.833041 / 4.584777 (-3.751735) | 4.665915 / 3.745712 (0.920203) | 4.543559 / 5.269862 (-0.726302) | 2.246926 / 4.565676 (-2.318750) | 0.098490 / 0.424275 (-0.325785) | 0.014934 / 0.007607 (0.007327) | 0.591878 / 0.226044 (0.365834) | 6.039852 / 2.268929 (3.770923) | 2.881244 / 55.444624 (-52.563381) | 2.486297 / 6.876477 (-4.390179) | 2.564642 / 2.142072 (0.422569) | 0.985684 / 4.805227 (-3.819543) | 0.199101 / 6.500664 (-6.301563) | 0.078138 / 0.075469 (0.002669) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.647744 / 1.841788 (-0.194043) | 18.986464 / 8.074308 (10.912156) | 17.246575 / 10.191392 (7.055183) | 0.219151 / 0.680424 (-0.461273) | 0.022219 / 0.534201 (-0.511982) | 0.547207 / 0.579283 (-0.032076) | 0.525943 / 0.434364 (0.091579) | 0.616909 / 0.540337 (0.076572) | 0.757423 / 1.386936 (-0.629513) |

</details>
</details>

![](https://cml.dev/watermark.png#f423b69cd4371bd03bb819c60450534f8850ad61 "CML watermark")