Set `usedforsecurity=False` in hashlib methods (FIPS compliance)
Opened by Wauplin (closed)

Related to https://github.com/huggingface/transformers/issues/27034 and https://github.com/huggingface/huggingface_hub/pull/1782.

**TL;DR:** `hashlib` is not a secure library for cryptography-related stuff. We are only using `hashlib` for non-security-related purposes in `datasets` so it's fine. From Python 3.9 we set can `usedforsecurity=False` in any `hashlib` method which is mandatory for companies that forbid the use of `hashlib` for security purposes. This PR fixes that.

**Note:** before merging this we need to release a new tokenizers version that would allow the newest `huggingface_hub` version (see https://github.com/huggingface/tokenizers/pull/1385). Otherwise it might create friction to users that want to install `datasets` + `tokenizers` at the same time.

Comments:
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008434 / 0.011353 (-0.002919) | 0.006755 / 0.011008 (-0.004253) | 0.106169 / 0.038508 (0.067661) | 0.049329 / 0.023109 (0.026220) | 0.433610 / 0.275898 (0.157712) | 0.441993 / 0.323480 (0.118513) | 0.004703 / 0.007986 (-0.003282) | 0.006996 / 0.004328 (0.002667) | 0.080330 / 0.004250 (0.076080) | 0.066098 / 0.037052 (0.029045) | 0.435444 / 0.258489 (0.176955) | 0.490442 / 0.293841 (0.196601) | 0.047050 / 0.128546 (-0.081496) | 0.014520 / 0.075646 (-0.061127) | 0.339805 / 0.419271 (-0.079467) | 0.101161 / 0.043533 (0.057629) | 0.423236 / 0.255139 (0.168097) | 0.455627 / 0.283200 (0.172427) | 0.036218 / 0.141683 (-0.105465) | 1.766128 / 1.452155 (0.313973) | 1.923919 / 1.492716 (0.431203) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.242939 / 0.018006 (0.224933) | 0.515582 / 0.000490 (0.515093) | 0.020271 / 0.000200 (0.020071) | 0.000383 / 0.000054 (0.000328) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030927 / 0.037411 (-0.006484) | 0.093951 / 0.014526 (0.079425) | 0.109028 / 0.176557 (-0.067529) | 0.174947 / 0.737135 (-0.562188) | 0.120538 / 0.296338 (-0.175800) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.553884 / 0.215209 (0.338675) | 5.424566 / 2.077655 (3.346911) | 2.439420 / 1.504120 (0.935301) | 2.019324 / 1.541195 (0.478129) | 2.170781 / 1.468490 (0.702290) | 0.924424 / 4.584777 (-3.660353) | 5.706029 / 3.745712 (1.960317) | 5.096911 / 5.269862 (-0.172951) | 3.168261 / 4.565676 (-1.397416) | 0.094336 / 0.424275 (-0.329940) | 0.015899 / 0.007607 (0.008292) | 0.709684 / 0.226044 (0.483639) | 7.476865 / 2.268929 (5.207936) | 3.350983 / 55.444624 (-52.093641) | 2.653419 / 6.876477 (-4.223058) | 2.802201 / 2.142072 (0.660129) | 1.081442 / 4.805227 (-3.723785) | 0.217025 / 6.500664 (-6.283639) | 0.077248 / 0.075469 (0.001779) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.598621 / 1.841788 (-0.243167) | 23.490338 / 8.074308 (15.416030) | 21.853488 / 10.191392 (11.662096) | 0.209625 / 0.680424 (-0.470799) | 0.028166 / 0.534201 (-0.506035) | 0.473883 / 0.579283 (-0.105400) | 0.584226 / 0.434364 (0.149862) | 0.538605 / 0.540337 (-0.001732) | 0.837060 / 1.386936 (-0.549876) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009029 / 0.011353 (-0.002324) | 0.004945 / 0.011008 (-0.006063) | 0.084539 / 0.038508 (0.046031) | 0.081014 / 0.023109 (0.057905) | 0.431291 / 0.275898 (0.155393) | 0.478913 / 0.323480 (0.155433) | 0.006107 / 0.007986 (-0.001879) | 0.003939 / 0.004328 (-0.000390) | 0.079932 / 0.004250 (0.075682) | 0.057936 / 0.037052 (0.020884) | 0.437295 / 0.258489 (0.178806) | 0.489790 / 0.293841 (0.195949) | 0.049544 / 0.128546 (-0.079003) | 0.013675 / 0.075646 (-0.061972) | 0.093143 / 0.419271 (-0.326128) | 0.064104 / 0.043533 (0.020571) | 0.444699 / 0.255139 (0.189560) | 0.443688 / 0.283200 (0.160489) | 0.034331 / 0.141683 (-0.107352) | 1.753014 / 1.452155 (0.300859) | 1.877274 / 1.492716 (0.384558) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.250460 / 0.018006 (0.232454) | 0.527241 / 0.000490 (0.526752) | 0.007679 / 0.000200 (0.007479) | 0.000115 / 0.000054 (0.000061) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033269 / 0.037411 (-0.004142) | 0.111262 / 0.014526 (0.096736) | 0.133503 / 0.176557 (-0.043053) | 0.177998 / 0.737135 (-0.559137) | 0.117899 / 0.296338 (-0.178440) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.633588 / 0.215209 (0.418379) | 6.105283 / 2.077655 (4.027628) | 2.779309 / 1.504120 (1.275189) | 2.445788 / 1.541195 (0.904594) | 2.396443 / 1.468490 (0.927953) | 0.925928 / 4.584777 (-3.658849) | 5.266142 / 3.745712 (1.520430) | 4.868830 / 5.269862 (-0.401031) | 2.998768 / 4.565676 (-1.566909) | 0.103135 / 0.424275 (-0.321140) | 0.008059 / 0.007607 (0.000452) | 0.753159 / 0.226044 (0.527115) | 7.532170 / 2.268929 (5.263242) | 3.563941 / 55.444624 (-51.880683) | 2.829208 / 6.876477 (-4.047269) | 2.913954 / 2.142072 (0.771881) | 1.085843 / 4.805227 (-3.719384) | 0.214195 / 6.500664 (-6.286469) | 0.071509 / 0.075469 (-0.003960) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.544819 / 1.841788 (-0.296968) | 23.790149 / 8.074308 (15.715841) | 23.086019 / 10.191392 (12.894627) | 0.242695 / 0.680424 (-0.437729) | 0.041706 / 0.534201 (-0.492495) | 0.552402 / 0.579283 (-0.026881) | 0.652518 / 0.434364 (0.218154) | 0.581876 / 0.540337 (0.041539) | 0.795425 / 1.386936 (-0.591511) |

</details>
</details>

![](https://cml.dev/watermark.png#117fdfccc8523fe150521ad74e478459fe2f297c "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004573 / 0.011353 (-0.006780) | 0.002965 / 0.011008 (-0.008043) | 0.061913 / 0.038508 (0.023405) | 0.029474 / 0.023109 (0.006365) | 0.258117 / 0.275898 (-0.017781) | 0.279854 / 0.323480 (-0.043626) | 0.003954 / 0.007986 (-0.004031) | 0.002479 / 0.004328 (-0.001850) | 0.048685 / 0.004250 (0.044434) | 0.044733 / 0.037052 (0.007681) | 0.256659 / 0.258489 (-0.001830) | 0.285235 / 0.293841 (-0.008606) | 0.023566 / 0.128546 (-0.104981) | 0.007291 / 0.075646 (-0.068355) | 0.202701 / 0.419271 (-0.216570) | 0.055706 / 0.043533 (0.012173) | 0.258790 / 0.255139 (0.003651) | 0.278675 / 0.283200 (-0.004525) | 0.018574 / 0.141683 (-0.123109) | 1.109359 / 1.452155 (-0.342796) | 1.184434 / 1.492716 (-0.308282) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095048 / 0.018006 (0.077042) | 0.305027 / 0.000490 (0.304537) | 0.000310 / 0.000200 (0.000110) | 0.000066 / 0.000054 (0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018183 / 0.037411 (-0.019228) | 0.066130 / 0.014526 (0.051604) | 0.073948 / 0.176557 (-0.102608) | 0.120458 / 0.737135 (-0.616678) | 0.075995 / 0.296338 (-0.220343) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.279419 / 0.215209 (0.064210) | 2.728591 / 2.077655 (0.650936) | 1.439016 / 1.504120 (-0.065104) | 1.325798 / 1.541195 (-0.215397) | 1.352050 / 1.468490 (-0.116440) | 0.395041 / 4.584777 (-4.189736) | 2.377651 / 3.745712 (-1.368061) | 2.618473 / 5.269862 (-2.651389) | 1.587580 / 4.565676 (-2.978096) | 0.045910 / 0.424275 (-0.378365) | 0.004843 / 0.007607 (-0.002764) | 0.335491 / 0.226044 (0.109447) | 3.378441 / 2.268929 (1.109512) | 1.827757 / 55.444624 (-53.616868) | 1.502360 / 6.876477 (-5.374117) | 1.508460 / 2.142072 (-0.633612) | 0.471309 / 4.805227 (-4.333918) | 0.098934 / 6.500664 (-6.401730) | 0.041705 / 0.075469 (-0.033764) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.945067 / 1.841788 (-0.896720) | 11.548209 / 8.074308 (3.473900) | 10.422628 / 10.191392 (0.231236) | 0.141494 / 0.680424 (-0.538929) | 0.014345 / 0.534201 (-0.519856) | 0.267750 / 0.579283 (-0.311533) | 0.261488 / 0.434364 (-0.172876) | 0.307192 / 0.540337 (-0.233145) | 0.427926 / 1.386936 (-0.959010) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004831 / 0.011353 (-0.006522) | 0.002876 / 0.011008 (-0.008132) | 0.048629 / 0.038508 (0.010121) | 0.055090 / 0.023109 (0.031981) | 0.271381 / 0.275898 (-0.004517) | 0.292350 / 0.323480 (-0.031130) | 0.004001 / 0.007986 (-0.003985) | 0.002389 / 0.004328 (-0.001939) | 0.047527 / 0.004250 (0.043277) | 0.038065 / 0.037052 (0.001012) | 0.277387 / 0.258489 (0.018898) | 0.307209 / 0.293841 (0.013368) | 0.025136 / 0.128546 (-0.103411) | 0.007309 / 0.075646 (-0.068338) | 0.054483 / 0.419271 (-0.364789) | 0.032807 / 0.043533 (-0.010726) | 0.274364 / 0.255139 (0.019225) | 0.290280 / 0.283200 (0.007080) | 0.017855 / 0.141683 (-0.123828) | 1.185912 / 1.452155 (-0.266243) | 1.228141 / 1.492716 (-0.264576) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094787 / 0.018006 (0.076781) | 0.314191 / 0.000490 (0.313701) | 0.000217 / 0.000200 (0.000017) | 0.000058 / 0.000054 (0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.020920 / 0.037411 (-0.016491) | 0.070446 / 0.014526 (0.055920) | 0.081371 / 0.176557 (-0.095186) | 0.119127 / 0.737135 (-0.618009) | 0.085658 / 0.296338 (-0.210680) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.290601 / 0.215209 (0.075392) | 2.874091 / 2.077655 (0.796436) | 1.598934 / 1.504120 (0.094814) | 1.464329 / 1.541195 (-0.076866) | 1.504943 / 1.468490 (0.036453) | 0.410457 / 4.584777 (-4.174320) | 2.428706 / 3.745712 (-1.317006) | 2.596510 / 5.269862 (-2.673352) | 1.547084 / 4.565676 (-3.018592) | 0.047546 / 0.424275 (-0.376729) | 0.004740 / 0.007607 (-0.002867) | 0.351168 / 0.226044 (0.125123) | 3.424554 / 2.268929 (1.155626) | 1.969792 / 55.444624 (-53.474832) | 1.676731 / 6.876477 (-5.199745) | 1.668769 / 2.142072 (-0.473304) | 0.482486 / 4.805227 (-4.322741) | 0.100018 / 6.500664 (-6.400646) | 0.040956 / 0.075469 (-0.034513) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.966306 / 1.841788 (-0.875482) | 12.158909 / 8.074308 (4.084601) | 10.926447 / 10.191392 (0.735055) | 0.130359 / 0.680424 (-0.550065) | 0.016162 / 0.534201 (-0.518039) | 0.269977 / 0.579283 (-0.309306) | 0.283366 / 0.434364 (-0.150997) | 0.304517 / 0.540337 (-0.235821) | 0.410398 / 1.386936 (-0.976539) |

</details>
</details>

![](https://cml.dev/watermark.png#53d5d6e57913465c22bb8074b0c0f968252cb12b "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004686 / 0.011353 (-0.006667) | 0.002764 / 0.011008 (-0.008244) | 0.061411 / 0.038508 (0.022902) | 0.030450 / 0.023109 (0.007341) | 0.247648 / 0.275898 (-0.028250) | 0.278033 / 0.323480 (-0.045447) | 0.002903 / 0.007986 (-0.005082) | 0.002350 / 0.004328 (-0.001979) | 0.047514 / 0.004250 (0.043264) | 0.044446 / 0.037052 (0.007393) | 0.256170 / 0.258489 (-0.002319) | 0.285977 / 0.293841 (-0.007864) | 0.023407 / 0.128546 (-0.105139) | 0.007223 / 0.075646 (-0.068423) | 0.201274 / 0.419271 (-0.217997) | 0.054022 / 0.043533 (0.010489) | 0.253841 / 0.255139 (-0.001298) | 0.278219 / 0.283200 (-0.004980) | 0.017796 / 0.141683 (-0.123886) | 1.105950 / 1.452155 (-0.346205) | 1.182021 / 1.492716 (-0.310695) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.089584 / 0.018006 (0.071578) | 0.299338 / 0.000490 (0.298849) | 0.000202 / 0.000200 (0.000003) | 0.000050 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018974 / 0.037411 (-0.018437) | 0.062352 / 0.014526 (0.047826) | 0.073667 / 0.176557 (-0.102889) | 0.119225 / 0.737135 (-0.617911) | 0.075393 / 0.296338 (-0.220945) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.282749 / 0.215209 (0.067540) | 2.795822 / 2.077655 (0.718167) | 1.492946 / 1.504120 (-0.011174) | 1.382340 / 1.541195 (-0.158855) | 1.377281 / 1.468490 (-0.091209) | 0.397361 / 4.584777 (-4.187415) | 2.379416 / 3.745712 (-1.366296) | 2.552967 / 5.269862 (-2.716895) | 1.546347 / 4.565676 (-3.019330) | 0.045851 / 0.424275 (-0.378424) | 0.004830 / 0.007607 (-0.002777) | 0.351194 / 0.226044 (0.125150) | 3.407406 / 2.268929 (1.138478) | 1.852983 / 55.444624 (-53.591641) | 1.536381 / 6.876477 (-5.340095) | 1.542786 / 2.142072 (-0.599287) | 0.471960 / 4.805227 (-4.333267) | 0.098336 / 6.500664 (-6.402328) | 0.041569 / 0.075469 (-0.033900) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.912718 / 1.841788 (-0.929070) | 11.339404 / 8.074308 (3.265095) | 10.480593 / 10.191392 (0.289201) | 0.139508 / 0.680424 (-0.540916) | 0.014210 / 0.534201 (-0.519991) | 0.268152 / 0.579283 (-0.311131) | 0.260503 / 0.434364 (-0.173860) | 0.304735 / 0.540337 (-0.235602) | 0.422155 / 1.386936 (-0.964781) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004714 / 0.011353 (-0.006638) | 0.002638 / 0.011008 (-0.008370) | 0.047967 / 0.038508 (0.009459) | 0.050758 / 0.023109 (0.027649) | 0.265619 / 0.275898 (-0.010279) | 0.286920 / 0.323480 (-0.036560) | 0.003936 / 0.007986 (-0.004050) | 0.002351 / 0.004328 (-0.001977) | 0.047642 / 0.004250 (0.043392) | 0.038412 / 0.037052 (0.001360) | 0.269561 / 0.258489 (0.011072) | 0.302057 / 0.293841 (0.008216) | 0.023893 / 0.128546 (-0.104653) | 0.006793 / 0.075646 (-0.068854) | 0.053091 / 0.419271 (-0.366180) | 0.032228 / 0.043533 (-0.011305) | 0.267110 / 0.255139 (0.011971) | 0.287211 / 0.283200 (0.004011) | 0.017945 / 0.141683 (-0.123738) | 1.191770 / 1.452155 (-0.260384) | 1.269644 / 1.492716 (-0.223072) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.088067 / 0.018006 (0.070061) | 0.298383 / 0.000490 (0.297893) | 0.000202 / 0.000200 (0.000002) | 0.000048 / 0.000054 (-0.000007) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.020685 / 0.037411 (-0.016726) | 0.069883 / 0.014526 (0.055357) | 0.080107 / 0.176557 (-0.096450) | 0.119311 / 0.737135 (-0.617825) | 0.080791 / 0.296338 (-0.215548) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.295781 / 0.215209 (0.080572) | 2.905536 / 2.077655 (0.827881) | 1.579184 / 1.504120 (0.075064) | 1.475937 / 1.541195 (-0.065258) | 1.533708 / 1.468490 (0.065218) | 0.409851 / 4.584777 (-4.174926) | 2.443217 / 3.745712 (-1.302496) | 2.543980 / 5.269862 (-2.725882) | 1.512187 / 4.565676 (-3.053489) | 0.046390 / 0.424275 (-0.377885) | 0.004762 / 0.007607 (-0.002845) | 0.345066 / 0.226044 (0.119021) | 3.485133 / 2.268929 (1.216204) | 1.954690 / 55.444624 (-53.489934) | 1.671104 / 6.876477 (-5.205372) | 1.655330 / 2.142072 (-0.486743) | 0.487910 / 4.805227 (-4.317317) | 0.097707 / 6.500664 (-6.402957) | 0.040379 / 0.075469 (-0.035090) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.981620 / 1.841788 (-0.860168) | 11.806530 / 8.074308 (3.732222) | 10.868275 / 10.191392 (0.676883) | 0.141230 / 0.680424 (-0.539194) | 0.015785 / 0.534201 (-0.518416) | 0.271416 / 0.579283 (-0.307867) | 0.276048 / 0.434364 (-0.158316) | 0.310988 / 0.540337 (-0.229349) | 0.410078 / 1.386936 (-0.976858) |

</details>
</details>

![](https://cml.dev/watermark.png#ec565740dee10c466ade16f81dee2783e442ba55 "CML watermark")


_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004803 / 0.011353 (-0.006550) | 0.002961 / 0.011008 (-0.008047) | 0.061431 / 0.038508 (0.022923) | 0.030189 / 0.023109 (0.007080) | 0.255755 / 0.275898 (-0.020143) | 0.277841 / 0.323480 (-0.045639) | 0.003083 / 0.007986 (-0.004902) | 0.002432 / 0.004328 (-0.001896) | 0.047674 / 0.004250 (0.043424) | 0.045066 / 0.037052 (0.008014) | 0.268701 / 0.258489 (0.010211) | 0.286673 / 0.293841 (-0.007168) | 0.023663 / 0.128546 (-0.104883) | 0.007148 / 0.075646 (-0.068499) | 0.201962 / 0.419271 (-0.217310) | 0.054953 / 0.043533 (0.011420) | 0.257155 / 0.255139 (0.002016) | 0.277769 / 0.283200 (-0.005431) | 0.017803 / 0.141683 (-0.123880) | 1.100270 / 1.452155 (-0.351884) | 1.146975 / 1.492716 (-0.345741) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092776 / 0.018006 (0.074770) | 0.303786 / 0.000490 (0.303296) | 0.000237 / 0.000200 (0.000037) | 0.000055 / 0.000054 (0.000000) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019647 / 0.037411 (-0.017765) | 0.063211 / 0.014526 (0.048686) | 0.076684 / 0.176557 (-0.099873) | 0.121952 / 0.737135 (-0.615184) | 0.077202 / 0.296338 (-0.219137) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.282087 / 0.215209 (0.066878) | 2.789204 / 2.077655 (0.711550) | 1.510376 / 1.504120 (0.006256) | 1.384241 / 1.541195 (-0.156954) | 1.414949 / 1.468490 (-0.053541) | 0.402206 / 4.584777 (-4.182570) | 2.377601 / 3.745712 (-1.368111) | 2.585354 / 5.269862 (-2.684508) | 1.592937 / 4.565676 (-2.972740) | 0.045217 / 0.424275 (-0.379058) | 0.004772 / 0.007607 (-0.002835) | 0.339584 / 0.226044 (0.113539) | 3.373184 / 2.268929 (1.104256) | 1.855196 / 55.444624 (-53.589428) | 1.599559 / 6.876477 (-5.276918) | 1.604421 / 2.142072 (-0.537651) | 0.467754 / 4.805227 (-4.337474) | 0.098244 / 6.500664 (-6.402420) | 0.042631 / 0.075469 (-0.032838) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.947680 / 1.841788 (-0.894108) | 11.539875 / 8.074308 (3.465567) | 10.340830 / 10.191392 (0.149438) | 0.145591 / 0.680424 (-0.534833) | 0.014367 / 0.534201 (-0.519834) | 0.270506 / 0.579283 (-0.308777) | 0.268825 / 0.434364 (-0.165539) | 0.308372 / 0.540337 (-0.231966) | 0.425039 / 1.386936 (-0.961897) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004813 / 0.011353 (-0.006540) | 0.002931 / 0.011008 (-0.008078) | 0.047997 / 0.038508 (0.009489) | 0.050753 / 0.023109 (0.027644) | 0.272704 / 0.275898 (-0.003194) | 0.294045 / 0.323480 (-0.029435) | 0.004059 / 0.007986 (-0.003927) | 0.002491 / 0.004328 (-0.001838) | 0.047621 / 0.004250 (0.043371) | 0.038824 / 0.037052 (0.001772) | 0.275322 / 0.258489 (0.016833) | 0.306447 / 0.293841 (0.012606) | 0.024402 / 0.128546 (-0.104145) | 0.007252 / 0.075646 (-0.068394) | 0.053346 / 0.419271 (-0.365925) | 0.032224 / 0.043533 (-0.011309) | 0.271468 / 0.255139 (0.016329) | 0.289429 / 0.283200 (0.006229) | 0.018285 / 0.141683 (-0.123398) | 1.116743 / 1.452155 (-0.335412) | 1.182724 / 1.492716 (-0.309993) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091899 / 0.018006 (0.073893) | 0.299161 / 0.000490 (0.298671) | 0.000224 / 0.000200 (0.000024) | 0.000053 / 0.000054 (-0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021823 / 0.037411 (-0.015588) | 0.071227 / 0.014526 (0.056701) | 0.080503 / 0.176557 (-0.096053) | 0.120243 / 0.737135 (-0.616892) | 0.082328 / 0.296338 (-0.214010) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.324951 / 0.215209 (0.109742) | 2.842358 / 2.077655 (0.764703) | 1.602317 / 1.504120 (0.098197) | 1.481103 / 1.541195 (-0.060091) | 1.497557 / 1.468490 (0.029067) | 0.406523 / 4.584777 (-4.178254) | 2.402743 / 3.745712 (-1.342970) | 2.545435 / 5.269862 (-2.724427) | 1.534071 / 4.565676 (-3.031605) | 0.046914 / 0.424275 (-0.377361) | 0.004728 / 0.007607 (-0.002879) | 0.341544 / 0.226044 (0.115499) | 3.412017 / 2.268929 (1.143089) | 1.937442 / 55.444624 (-53.507182) | 1.668774 / 6.876477 (-5.207703) | 1.668908 / 2.142072 (-0.473165) | 0.477398 / 4.805227 (-4.327829) | 0.098531 / 6.500664 (-6.402133) | 0.041077 / 0.075469 (-0.034392) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.983888 / 1.841788 (-0.857900) | 12.072703 / 8.074308 (3.998395) | 11.028622 / 10.191392 (0.837230) | 0.148097 / 0.680424 (-0.532327) | 0.015869 / 0.534201 (-0.518332) | 0.267609 / 0.579283 (-0.311674) | 0.272345 / 0.434364 (-0.162019) | 0.303840 / 0.540337 (-0.236497) | 0.409199 / 1.386936 (-0.977737) |

</details>
</details>

![](https://cml.dev/watermark.png#1487df064580bd23458234fab2e85876d9364e03 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005016 / 0.011353 (-0.006337) | 0.002931 / 0.011008 (-0.008077) | 0.062142 / 0.038508 (0.023634) | 0.030758 / 0.023109 (0.007648) | 0.251689 / 0.275898 (-0.024209) | 0.272114 / 0.323480 (-0.051366) | 0.004102 / 0.007986 (-0.003884) | 0.002500 / 0.004328 (-0.001828) | 0.049187 / 0.004250 (0.044937) | 0.047150 / 0.037052 (0.010098) | 0.256497 / 0.258489 (-0.001992) | 0.288069 / 0.293841 (-0.005772) | 0.023915 / 0.128546 (-0.104632) | 0.007204 / 0.075646 (-0.068442) | 0.204257 / 0.419271 (-0.215015) | 0.063879 / 0.043533 (0.020346) | 0.253008 / 0.255139 (-0.002131) | 0.266554 / 0.283200 (-0.016645) | 0.018929 / 0.141683 (-0.122754) | 1.140547 / 1.452155 (-0.311608) | 1.197049 / 1.492716 (-0.295668) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094111 / 0.018006 (0.076105) | 0.301618 / 0.000490 (0.301128) | 0.000219 / 0.000200 (0.000019) | 0.000042 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018614 / 0.037411 (-0.018797) | 0.062426 / 0.014526 (0.047900) | 0.073079 / 0.176557 (-0.103477) | 0.120313 / 0.737135 (-0.616823) | 0.076445 / 0.296338 (-0.219894) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.285151 / 0.215209 (0.069942) | 2.754272 / 2.077655 (0.676617) | 1.485254 / 1.504120 (-0.018866) | 1.368412 / 1.541195 (-0.172783) | 1.402819 / 1.468490 (-0.065671) | 0.396561 / 4.584777 (-4.188216) | 2.375708 / 3.745712 (-1.370004) | 2.656088 / 5.269862 (-2.613773) | 1.588676 / 4.565676 (-2.977001) | 0.048662 / 0.424275 (-0.375613) | 0.004963 / 0.007607 (-0.002644) | 0.339747 / 0.226044 (0.113702) | 3.315841 / 2.268929 (1.046912) | 1.841439 / 55.444624 (-53.603186) | 1.547803 / 6.876477 (-5.328674) | 1.601872 / 2.142072 (-0.540200) | 0.468637 / 4.805227 (-4.336591) | 0.099423 / 6.500664 (-6.401241) | 0.041926 / 0.075469 (-0.033543) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.933058 / 1.841788 (-0.908730) | 11.680870 / 8.074308 (3.606561) | 10.239009 / 10.191392 (0.047617) | 0.129974 / 0.680424 (-0.550450) | 0.014081 / 0.534201 (-0.520120) | 0.273076 / 0.579283 (-0.306207) | 0.261914 / 0.434364 (-0.172450) | 0.305982 / 0.540337 (-0.234356) | 0.430623 / 1.386936 (-0.956313) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004969 / 0.011353 (-0.006384) | 0.003084 / 0.011008 (-0.007924) | 0.048686 / 0.038508 (0.010178) | 0.057234 / 0.023109 (0.034125) | 0.295408 / 0.275898 (0.019510) | 0.323774 / 0.323480 (0.000294) | 0.004014 / 0.007986 (-0.003972) | 0.002423 / 0.004328 (-0.001905) | 0.048000 / 0.004250 (0.043749) | 0.039872 / 0.037052 (0.002820) | 0.294717 / 0.258489 (0.036228) | 0.331149 / 0.293841 (0.037309) | 0.027884 / 0.128546 (-0.100662) | 0.007155 / 0.075646 (-0.068491) | 0.053812 / 0.419271 (-0.365460) | 0.032483 / 0.043533 (-0.011050) | 0.293402 / 0.255139 (0.038263) | 0.312553 / 0.283200 (0.029354) | 0.017848 / 0.141683 (-0.123835) | 1.125600 / 1.452155 (-0.326554) | 1.189469 / 1.492716 (-0.303248) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096198 / 0.018006 (0.078191) | 0.305096 / 0.000490 (0.304607) | 0.000229 / 0.000200 (0.000029) | 0.000045 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021992 / 0.037411 (-0.015419) | 0.072082 / 0.014526 (0.057556) | 0.082704 / 0.176557 (-0.093853) | 0.124512 / 0.737135 (-0.612624) | 0.084541 / 0.296338 (-0.211797) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.296440 / 0.215209 (0.081231) | 2.923392 / 2.077655 (0.845738) | 1.599057 / 1.504120 (0.094937) | 1.480473 / 1.541195 (-0.060722) | 1.551837 / 1.468490 (0.083347) | 0.418618 / 4.584777 (-4.166159) | 2.472727 / 3.745712 (-1.272985) | 2.796141 / 5.269862 (-2.473721) | 1.629139 / 4.565676 (-2.936538) | 0.047703 / 0.424275 (-0.376572) | 0.004971 / 0.007607 (-0.002636) | 0.354453 / 0.226044 (0.128408) | 3.514861 / 2.268929 (1.245932) | 1.993597 / 55.444624 (-53.451028) | 1.694386 / 6.876477 (-5.182090) | 1.748562 / 2.142072 (-0.393510) | 0.487158 / 4.805227 (-4.318070) | 0.102021 / 6.500664 (-6.398643) | 0.042648 / 0.075469 (-0.032821) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.974950 / 1.841788 (-0.866837) | 13.391204 / 8.074308 (5.316896) | 11.474696 / 10.191392 (1.283304) | 0.142618 / 0.680424 (-0.537806) | 0.016163 / 0.534201 (-0.518038) | 0.271453 / 0.579283 (-0.307830) | 0.287049 / 0.434364 (-0.147315) | 0.309069 / 0.540337 (-0.231268) | 0.417117 / 1.386936 (-0.969819) |

</details>
</details>

![](https://cml.dev/watermark.png#35a3422cfcebfef5b09ae70c22843ffadaf44c46 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004974 / 0.011353 (-0.006379) | 0.002950 / 0.011008 (-0.008058) | 0.061856 / 0.038508 (0.023348) | 0.030539 / 0.023109 (0.007429) | 0.250105 / 0.275898 (-0.025793) | 0.276687 / 0.323480 (-0.046793) | 0.003077 / 0.007986 (-0.004908) | 0.002412 / 0.004328 (-0.001916) | 0.048336 / 0.004250 (0.044086) | 0.045849 / 0.037052 (0.008797) | 0.251757 / 0.258489 (-0.006732) | 0.284914 / 0.293841 (-0.008927) | 0.024033 / 0.128546 (-0.104513) | 0.007343 / 0.075646 (-0.068303) | 0.202867 / 0.419271 (-0.216405) | 0.061294 / 0.043533 (0.017762) | 0.263590 / 0.255139 (0.008451) | 0.272744 / 0.283200 (-0.010455) | 0.019613 / 0.141683 (-0.122070) | 1.104263 / 1.452155 (-0.347892) | 1.164128 / 1.492716 (-0.328588) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094261 / 0.018006 (0.076255) | 0.303340 / 0.000490 (0.302850) | 0.000215 / 0.000200 (0.000015) | 0.000057 / 0.000054 (0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018381 / 0.037411 (-0.019030) | 0.062727 / 0.014526 (0.048201) | 0.074955 / 0.176557 (-0.101602) | 0.124810 / 0.737135 (-0.612326) | 0.074335 / 0.296338 (-0.222004) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.279368 / 0.215209 (0.064159) | 2.721641 / 2.077655 (0.643986) | 1.510773 / 1.504120 (0.006653) | 1.364349 / 1.541195 (-0.176845) | 1.386044 / 1.468490 (-0.082446) | 0.403051 / 4.584777 (-4.181726) | 2.416525 / 3.745712 (-1.329187) | 2.623198 / 5.269862 (-2.646663) | 1.560869 / 4.565676 (-3.004808) | 0.046613 / 0.424275 (-0.377662) | 0.004861 / 0.007607 (-0.002746) | 0.337875 / 0.226044 (0.111830) | 3.289956 / 2.268929 (1.021028) | 1.851707 / 55.444624 (-53.592917) | 1.571092 / 6.876477 (-5.305385) | 1.600328 / 2.142072 (-0.541745) | 0.480766 / 4.805227 (-4.324461) | 0.099138 / 6.500664 (-6.401526) | 0.041691 / 0.075469 (-0.033779) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.941162 / 1.841788 (-0.900626) | 11.745335 / 8.074308 (3.671027) | 10.645509 / 10.191392 (0.454117) | 0.132506 / 0.680424 (-0.547918) | 0.015192 / 0.534201 (-0.519009) | 0.272483 / 0.579283 (-0.306800) | 0.270269 / 0.434364 (-0.164094) | 0.309580 / 0.540337 (-0.230758) | 0.431513 / 1.386936 (-0.955423) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005068 / 0.011353 (-0.006285) | 0.003069 / 0.011008 (-0.007939) | 0.048605 / 0.038508 (0.010097) | 0.059557 / 0.023109 (0.036448) | 0.275092 / 0.275898 (-0.000806) | 0.298910 / 0.323480 (-0.024570) | 0.004198 / 0.007986 (-0.003788) | 0.002499 / 0.004328 (-0.001830) | 0.048248 / 0.004250 (0.043997) | 0.040302 / 0.037052 (0.003249) | 0.279539 / 0.258489 (0.021050) | 0.312500 / 0.293841 (0.018659) | 0.025407 / 0.128546 (-0.103140) | 0.007364 / 0.075646 (-0.068282) | 0.053086 / 0.419271 (-0.366186) | 0.033291 / 0.043533 (-0.010242) | 0.276521 / 0.255139 (0.021382) | 0.292943 / 0.283200 (0.009743) | 0.019416 / 0.141683 (-0.122267) | 1.151734 / 1.452155 (-0.300421) | 1.205021 / 1.492716 (-0.287695) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094112 / 0.018006 (0.076106) | 0.309534 / 0.000490 (0.309044) | 0.000219 / 0.000200 (0.000019) | 0.000052 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021539 / 0.037411 (-0.015872) | 0.070325 / 0.014526 (0.055799) | 0.080468 / 0.176557 (-0.096089) | 0.121095 / 0.737135 (-0.616040) | 0.082008 / 0.296338 (-0.214331) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.302591 / 0.215209 (0.087382) | 2.943475 / 2.077655 (0.865820) | 1.597970 / 1.504120 (0.093850) | 1.468774 / 1.541195 (-0.072421) | 1.504812 / 1.468490 (0.036322) | 0.413715 / 4.584777 (-4.171062) | 2.418319 / 3.745712 (-1.327393) | 2.616656 / 5.269862 (-2.653206) | 1.558165 / 4.565676 (-3.007512) | 0.047169 / 0.424275 (-0.377106) | 0.004761 / 0.007607 (-0.002846) | 0.347225 / 0.226044 (0.121180) | 3.479624 / 2.268929 (1.210696) | 1.961253 / 55.444624 (-53.483371) | 1.673532 / 6.876477 (-5.202944) | 1.698900 / 2.142072 (-0.443172) | 0.488373 / 4.805227 (-4.316855) | 0.098322 / 6.500664 (-6.402342) | 0.040832 / 0.075469 (-0.034637) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.009133 / 1.841788 (-0.832655) | 13.373258 / 8.074308 (5.298949) | 11.327360 / 10.191392 (1.135968) | 0.135778 / 0.680424 (-0.544646) | 0.015813 / 0.534201 (-0.518388) | 0.275404 / 0.579283 (-0.303879) | 0.282564 / 0.434364 (-0.151799) | 0.311830 / 0.540337 (-0.228507) | 0.419008 / 1.386936 (-0.967928) |

</details>
</details>

![](https://cml.dev/watermark.png#4592709e5399f91b5b392f4fd73687985365c909 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004899 / 0.011353 (-0.006454) | 0.002780 / 0.011008 (-0.008229) | 0.061997 / 0.038508 (0.023489) | 0.029909 / 0.023109 (0.006800) | 0.233445 / 0.275898 (-0.042453) | 0.254128 / 0.323480 (-0.069351) | 0.002927 / 0.007986 (-0.005058) | 0.002396 / 0.004328 (-0.001932) | 0.048118 / 0.004250 (0.043868) | 0.044520 / 0.037052 (0.007468) | 0.237594 / 0.258489 (-0.020895) | 0.268407 / 0.293841 (-0.025434) | 0.023517 / 0.128546 (-0.105029) | 0.007035 / 0.075646 (-0.068612) | 0.202803 / 0.419271 (-0.216469) | 0.057692 / 0.043533 (0.014159) | 0.237058 / 0.255139 (-0.018081) | 0.252966 / 0.283200 (-0.030233) | 0.017934 / 0.141683 (-0.123748) | 1.096406 / 1.452155 (-0.355749) | 1.153509 / 1.492716 (-0.339207) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091812 / 0.018006 (0.073806) | 0.298410 / 0.000490 (0.297920) | 0.000228 / 0.000200 (0.000028) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018333 / 0.037411 (-0.019078) | 0.062685 / 0.014526 (0.048159) | 0.073295 / 0.176557 (-0.103261) | 0.119234 / 0.737135 (-0.617901) | 0.074603 / 0.296338 (-0.221736) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.279078 / 0.215209 (0.063869) | 2.768535 / 2.077655 (0.690880) | 1.457049 / 1.504120 (-0.047071) | 1.326870 / 1.541195 (-0.214325) | 1.349657 / 1.468490 (-0.118833) | 0.405003 / 4.584777 (-4.179774) | 2.428726 / 3.745712 (-1.316986) | 2.595776 / 5.269862 (-2.674086) | 1.557879 / 4.565676 (-3.007797) | 0.045985 / 0.424275 (-0.378291) | 0.004854 / 0.007607 (-0.002753) | 0.336437 / 0.226044 (0.110392) | 3.317330 / 2.268929 (1.048401) | 1.784525 / 55.444624 (-53.660100) | 1.500295 / 6.876477 (-5.376182) | 1.529869 / 2.142072 (-0.612203) | 0.473426 / 4.805227 (-4.331801) | 0.099609 / 6.500664 (-6.401055) | 0.042054 / 0.075469 (-0.033415) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.937154 / 1.841788 (-0.904633) | 11.482383 / 8.074308 (3.408075) | 10.468769 / 10.191392 (0.277377) | 0.132724 / 0.680424 (-0.547700) | 0.015242 / 0.534201 (-0.518959) | 0.281124 / 0.579283 (-0.298159) | 0.268603 / 0.434364 (-0.165761) | 0.311410 / 0.540337 (-0.228928) | 0.431817 / 1.386936 (-0.955119) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004695 / 0.011353 (-0.006658) | 0.002873 / 0.011008 (-0.008135) | 0.048133 / 0.038508 (0.009625) | 0.052505 / 0.023109 (0.029396) | 0.271679 / 0.275898 (-0.004219) | 0.292530 / 0.323480 (-0.030950) | 0.003844 / 0.007986 (-0.004142) | 0.002417 / 0.004328 (-0.001912) | 0.048619 / 0.004250 (0.044369) | 0.039152 / 0.037052 (0.002100) | 0.276575 / 0.258489 (0.018086) | 0.307836 / 0.293841 (0.013995) | 0.023877 / 0.128546 (-0.104669) | 0.006897 / 0.075646 (-0.068749) | 0.053241 / 0.419271 (-0.366031) | 0.032487 / 0.043533 (-0.011046) | 0.274205 / 0.255139 (0.019066) | 0.289701 / 0.283200 (0.006502) | 0.018250 / 0.141683 (-0.123432) | 1.137902 / 1.452155 (-0.314253) | 1.202043 / 1.492716 (-0.290673) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091453 / 0.018006 (0.073446) | 0.297032 / 0.000490 (0.296543) | 0.000224 / 0.000200 (0.000024) | 0.000056 / 0.000054 (0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021062 / 0.037411 (-0.016349) | 0.069848 / 0.014526 (0.055322) | 0.084337 / 0.176557 (-0.092219) | 0.119951 / 0.737135 (-0.617184) | 0.082805 / 0.296338 (-0.213533) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.297056 / 0.215209 (0.081846) | 2.890110 / 2.077655 (0.812456) | 1.609918 / 1.504120 (0.105798) | 1.491184 / 1.541195 (-0.050011) | 1.529433 / 1.468490 (0.060943) | 0.396081 / 4.584777 (-4.188696) | 2.408310 / 3.745712 (-1.337402) | 2.567905 / 5.269862 (-2.701957) | 1.514465 / 4.565676 (-3.051212) | 0.045329 / 0.424275 (-0.378946) | 0.004738 / 0.007607 (-0.002869) | 0.344373 / 0.226044 (0.118328) | 3.428333 / 2.268929 (1.159404) | 1.981401 / 55.444624 (-53.463223) | 1.688007 / 6.876477 (-5.188470) | 1.685542 / 2.142072 (-0.456531) | 0.478045 / 4.805227 (-4.327182) | 0.096664 / 6.500664 (-6.404001) | 0.040335 / 0.075469 (-0.035135) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.972912 / 1.841788 (-0.868876) | 12.055045 / 8.074308 (3.980737) | 10.821073 / 10.191392 (0.629681) | 0.139177 / 0.680424 (-0.541247) | 0.015046 / 0.534201 (-0.519155) | 0.275670 / 0.579283 (-0.303613) | 0.280366 / 0.434364 (-0.153998) | 0.315781 / 0.540337 (-0.224556) | 0.424536 / 1.386936 (-0.962400) |

</details>
</details>

![](https://cml.dev/watermark.png#0684b471d6ca8a235162f5575f624b6eda7956c5 "CML watermark")


I'm finally merging as `transformers`/`tokenizers` dependency pins have been removed + `huggingface_hub 0.19.4` has fixed the deps incompatibility issue. All good now :)

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004435 / 0.011353 (-0.006918) | 0.002924 / 0.011008 (-0.008084) | 0.062159 / 0.038508 (0.023651) | 0.029639 / 0.023109 (0.006529) | 0.237470 / 0.275898 (-0.038428) | 0.269641 / 0.323480 (-0.053839) | 0.004124 / 0.007986 (-0.003862) | 0.002528 / 0.004328 (-0.001800) | 0.048114 / 0.004250 (0.043864) | 0.046055 / 0.037052 (0.009002) | 0.245844 / 0.258489 (-0.012645) | 0.278085 / 0.293841 (-0.015756) | 0.023152 / 0.128546 (-0.105394) | 0.007194 / 0.075646 (-0.068452) | 0.206493 / 0.419271 (-0.212778) | 0.055687 / 0.043533 (0.012155) | 0.243301 / 0.255139 (-0.011838) | 0.267645 / 0.283200 (-0.015555) | 0.017413 / 0.141683 (-0.124270) | 1.113071 / 1.452155 (-0.339083) | 1.201436 / 1.492716 (-0.291280) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092576 / 0.018006 (0.074570) | 0.303516 / 0.000490 (0.303027) | 0.000213 / 0.000200 (0.000013) | 0.000043 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019108 / 0.037411 (-0.018303) | 0.062326 / 0.014526 (0.047800) | 0.073711 / 0.176557 (-0.102846) | 0.120414 / 0.737135 (-0.616721) | 0.075837 / 0.296338 (-0.220501) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.278267 / 0.215209 (0.063058) | 2.766231 / 2.077655 (0.688576) | 1.455613 / 1.504120 (-0.048507) | 1.337128 / 1.541195 (-0.204066) | 1.357659 / 1.468490 (-0.110831) | 0.404549 / 4.584777 (-4.180228) | 2.409084 / 3.745712 (-1.336628) | 2.645000 / 5.269862 (-2.624861) | 1.600475 / 4.565676 (-2.965201) | 0.046680 / 0.424275 (-0.377595) | 0.004887 / 0.007607 (-0.002720) | 0.340338 / 0.226044 (0.114294) | 3.332647 / 2.268929 (1.063719) | 1.852529 / 55.444624 (-53.592096) | 1.532442 / 6.876477 (-5.344035) | 1.550383 / 2.142072 (-0.591689) | 0.482702 / 4.805227 (-4.322525) | 0.101067 / 6.500664 (-6.399597) | 0.042132 / 0.075469 (-0.033337) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.945481 / 1.841788 (-0.896307) | 11.886240 / 8.074308 (3.811932) | 10.484620 / 10.191392 (0.293228) | 0.130906 / 0.680424 (-0.549518) | 0.014880 / 0.534201 (-0.519321) | 0.268836 / 0.579283 (-0.310447) | 0.268112 / 0.434364 (-0.166251) | 0.304300 / 0.540337 (-0.236038) | 0.440262 / 1.386936 (-0.946674) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005028 / 0.011353 (-0.006325) | 0.002937 / 0.011008 (-0.008071) | 0.049038 / 0.038508 (0.010530) | 0.057763 / 0.023109 (0.034653) | 0.273196 / 0.275898 (-0.002702) | 0.295519 / 0.323480 (-0.027961) | 0.004102 / 0.007986 (-0.003883) | 0.002487 / 0.004328 (-0.001841) | 0.049148 / 0.004250 (0.044898) | 0.040303 / 0.037052 (0.003251) | 0.279187 / 0.258489 (0.020698) | 0.311086 / 0.293841 (0.017245) | 0.024961 / 0.128546 (-0.103585) | 0.007264 / 0.075646 (-0.068382) | 0.055711 / 0.419271 (-0.363561) | 0.032355 / 0.043533 (-0.011178) | 0.274304 / 0.255139 (0.019165) | 0.290953 / 0.283200 (0.007753) | 0.018358 / 0.141683 (-0.123325) | 1.115984 / 1.452155 (-0.336170) | 1.190409 / 1.492716 (-0.302308) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095765 / 0.018006 (0.077759) | 0.287947 / 0.000490 (0.287457) | 0.000242 / 0.000200 (0.000042) | 0.000047 / 0.000054 (-0.000007) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022165 / 0.037411 (-0.015246) | 0.070465 / 0.014526 (0.055940) | 0.082078 / 0.176557 (-0.094479) | 0.120209 / 0.737135 (-0.616926) | 0.084573 / 0.296338 (-0.211765) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.298492 / 0.215209 (0.083283) | 2.924981 / 2.077655 (0.847327) | 1.597326 / 1.504120 (0.093206) | 1.459132 / 1.541195 (-0.082062) | 1.511471 / 1.468490 (0.042981) | 0.406671 / 4.584777 (-4.178106) | 2.443154 / 3.745712 (-1.302558) | 2.591131 / 5.269862 (-2.678731) | 1.549931 / 4.565676 (-3.015745) | 0.047042 / 0.424275 (-0.377234) | 0.004891 / 0.007607 (-0.002716) | 0.346274 / 0.226044 (0.120230) | 3.456050 / 2.268929 (1.187121) | 1.959328 / 55.444624 (-53.485296) | 1.647631 / 6.876477 (-5.228845) | 1.692024 / 2.142072 (-0.450049) | 0.478307 / 4.805227 (-4.326920) | 0.098738 / 6.500664 (-6.401926) | 0.041743 / 0.075469 (-0.033726) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.984619 / 1.841788 (-0.857168) | 12.403984 / 8.074308 (4.329676) | 10.974347 / 10.191392 (0.782955) | 0.132893 / 0.680424 (-0.547530) | 0.015504 / 0.534201 (-0.518697) | 0.275354 / 0.579283 (-0.303929) | 0.283312 / 0.434364 (-0.151052) | 0.313661 / 0.540337 (-0.226677) | 0.419065 / 1.386936 (-0.967871) |

</details>
</details>

![](https://cml.dev/watermark.png#c65315e4a8308f04fcb025039afe2a2e43b5684e "CML watermark")