Remove set_access_token usage + fail tests if FutureWarning
State: closed | Author: Wauplin (CONTRIBUTOR)

`set_access_token` is deprecated and will be removed in `huggingface_hub>=0.14`.

This PR removes it from the tests (it was not used in `datasets` source code itself). FYI, it was not needed since `set_access_token` was just setting git credentials and `datasets` doesn't seem to use git anywhere.

In the future, use `set_git_credential` if needed. It is a git-credential-agnostic helper, i.e. you can store your git token in `git-credential-cache`, `git-credential-store`, `osxkeychain`, etc. The legacy `set_access_token` could only set in `git-credential-store` no matter the user preference.

(for context, I found out about this while working on https://github.com/huggingface/huggingface_hub/pull/1381)

---

In addition to this, I have added

```
filterwarnings =
    error::FutureWarning:huggingface_hub*
```

to the `setup.cfg` config file to fail on future warnings from `huggingface_hub`. In `hfh`'s CI we trigger on FutureWarning from any package but it's less robust (any package update leads can lead to a failure). No obligation to keep it like that (I can remove it if you prefer) but I think it's a good idea in order to track future FutureWarnings.

FYI, in `huggingface_hub` tests we use `-Werror::FutureWarning --log-cli-level=INFO -sv --durations=0`
- FutureWarning are processed as error
- verbose mode / INFO logs (and above) are captured for easier debugging in github report
- track each test duration, just to see where we can improve. We have a quite long CI (~10min) so it helped improve that.

Comments: <details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008505 / 0.011353 (-0.002848) | 0.004445 / 0.011008 (-0.006563) | 0.102197 / 0.038508 (0.063689) | 0.029886 / 0.023109 (0.006776) | 0.305387 / 0.275898 (0.029489) | 0.355986 / 0.323480 (0.032507) | 0.006814 / 0.007986 (-0.001172) | 0.003298 / 0.004328 (-0.001030) | 0.079204 / 0.004250 (0.074954) | 0.035618 / 0.037052 (-0.001434) | 0.320430 / 0.258489 (0.061941) | 0.353330 / 0.293841 (0.059490) | 0.033280 / 0.128546 (-0.095266) | 0.011300 / 0.075646 (-0.064347) | 0.324627 / 0.419271 (-0.094644) | 0.040405 / 0.043533 (-0.003128) | 0.308760 / 0.255139 (0.053621) | 0.331885 / 0.283200 (0.048685) | 0.084605 / 0.141683 (-0.057077) | 1.576598 / 1.452155 (0.124443) | 1.530694 / 1.492716 (0.037977) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.191142 / 0.018006 (0.173136) | 0.404042 / 0.000490 (0.403552) | 0.001185 / 0.000200 (0.000985) | 0.000074 / 0.000054 (0.000019) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022889 / 0.037411 (-0.014523) | 0.095862 / 0.014526 (0.081336) | 0.104382 / 0.176557 (-0.072175) | 0.139407 / 0.737135 (-0.597728) | 0.106813 / 0.296338 (-0.189525) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.419083 / 0.215209 (0.203874) | 4.188702 / 2.077655 (2.111047) | 1.897854 / 1.504120 (0.393734) | 1.689544 / 1.541195 (0.148350) | 1.714032 / 1.468490 (0.245542) | 0.695541 / 4.584777 (-3.889236) | 3.370584 / 3.745712 (-0.375128) | 3.205549 / 5.269862 (-2.064313) | 1.641202 / 4.565676 (-2.924474) | 0.081849 / 0.424275 (-0.342426) | 0.012043 / 0.007607 (0.004436) | 0.529618 / 0.226044 (0.303574) | 5.314167 / 2.268929 (3.045238) | 2.357271 / 55.444624 (-53.087353) | 1.979684 / 6.876477 (-4.896793) | 2.030057 / 2.142072 (-0.112015) | 0.813013 / 4.805227 (-3.992214) | 0.150165 / 6.500664 (-6.350499) | 0.064595 / 0.075469 (-0.010874) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.237824 / 1.841788 (-0.603964) | 13.552178 / 8.074308 (5.477870) | 14.089433 / 10.191392 (3.898041) | 0.149325 / 0.680424 (-0.531099) | 0.028543 / 0.534201 (-0.505658) | 0.396848 / 0.579283 (-0.182435) | 0.396230 / 0.434364 (-0.038134) | 0.466317 / 0.540337 (-0.074021) | 0.539579 / 1.386936 (-0.847357) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006224 / 0.011353 (-0.005128) | 0.004429 / 0.011008 (-0.006579) | 0.075740 / 0.038508 (0.037232) | 0.026717 / 0.023109 (0.003608) | 0.341685 / 0.275898 (0.065787) | 0.383671 / 0.323480 (0.060191) | 0.004682 / 0.007986 (-0.003304) | 0.004681 / 0.004328 (0.000352) | 0.076638 / 0.004250 (0.072387) | 0.034577 / 0.037052 (-0.002476) | 0.341160 / 0.258489 (0.082671) | 0.407590 / 0.293841 (0.113749) | 0.031121 / 0.128546 (-0.097425) | 0.011479 / 0.075646 (-0.064167) | 0.085299 / 0.419271 (-0.333973) | 0.042005 / 0.043533 (-0.001528) | 0.339682 / 0.255139 (0.084543) | 0.377669 / 0.283200 (0.094469) | 0.087751 / 0.141683 (-0.053932) | 1.523910 / 1.452155 (0.071756) | 1.607487 / 1.492716 (0.114771) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.225605 / 0.018006 (0.207599) | 0.395851 / 0.000490 (0.395361) | 0.004404 / 0.000200 (0.004204) | 0.000082 / 0.000054 (0.000028) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024489 / 0.037411 (-0.012922) | 0.099813 / 0.014526 (0.085287) | 0.107392 / 0.176557 (-0.069165) | 0.139567 / 0.737135 (-0.597568) | 0.110080 / 0.296338 (-0.186258) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.449051 / 0.215209 (0.233841) | 4.463098 / 2.077655 (2.385443) | 2.122548 / 1.504120 (0.618428) | 1.913863 / 1.541195 (0.372669) | 1.963988 / 1.468490 (0.495498) | 0.698442 / 4.584777 (-3.886335) | 3.330425 / 3.745712 (-0.415287) | 1.867843 / 5.269862 (-3.402019) | 1.163740 / 4.565676 (-3.401937) | 0.083209 / 0.424275 (-0.341066) | 0.012594 / 0.007607 (0.004987) | 0.547074 / 0.226044 (0.321030) | 5.474779 / 2.268929 (3.205851) | 2.548025 / 55.444624 (-52.896599) | 2.202435 / 6.876477 (-4.674041) | 2.220330 / 2.142072 (0.078257) | 0.810104 / 4.805227 (-3.995124) | 0.151141 / 6.500664 (-6.349523) | 0.066204 / 0.075469 (-0.009265) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.272075 / 1.841788 (-0.569712) | 13.749523 / 8.074308 (5.675215) | 14.270974 / 10.191392 (4.079582) | 0.141285 / 0.680424 (-0.539139) | 0.016526 / 0.534201 (-0.517675) | 0.393175 / 0.579283 (-0.186109) | 0.391577 / 0.434364 (-0.042787) | 0.492824 / 0.540337 (-0.047513) | 0.580069 / 1.386936 (-0.806867) |

</details>
</details>

![](https://cml.dev/watermark.png#1cda14136c9f79c763c17d49b77eabfb233fbb35 "CML watermark")


_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008901 / 0.011353 (-0.002452) | 0.005017 / 0.011008 (-0.005991) | 0.099340 / 0.038508 (0.060832) | 0.034218 / 0.023109 (0.011109) | 0.295927 / 0.275898 (0.020029) | 0.330087 / 0.323480 (0.006607) | 0.008041 / 0.007986 (0.000056) | 0.005013 / 0.004328 (0.000685) | 0.074255 / 0.004250 (0.070004) | 0.049634 / 0.037052 (0.012582) | 0.299972 / 0.258489 (0.041483) | 0.349879 / 0.293841 (0.056038) | 0.038500 / 0.128546 (-0.090047) | 0.011980 / 0.075646 (-0.063666) | 0.332408 / 0.419271 (-0.086863) | 0.048385 / 0.043533 (0.004852) | 0.300393 / 0.255139 (0.045254) | 0.316972 / 0.283200 (0.033772) | 0.101674 / 0.141683 (-0.040009) | 1.424300 / 1.452155 (-0.027854) | 1.520658 / 1.492716 (0.027942) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.270084 / 0.018006 (0.252078) | 0.538612 / 0.000490 (0.538123) | 0.004439 / 0.000200 (0.004240) | 0.000089 / 0.000054 (0.000035) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026841 / 0.037411 (-0.010570) | 0.106454 / 0.014526 (0.091928) | 0.118371 / 0.176557 (-0.058186) | 0.155545 / 0.737135 (-0.581590) | 0.125119 / 0.296338 (-0.171220) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.395794 / 0.215209 (0.180585) | 3.958195 / 2.077655 (1.880540) | 1.789010 / 1.504120 (0.284890) | 1.601380 / 1.541195 (0.060186) | 1.641062 / 1.468490 (0.172572) | 0.679547 / 4.584777 (-3.905230) | 3.778018 / 3.745712 (0.032306) | 2.101232 / 5.269862 (-3.168630) | 1.463932 / 4.565676 (-3.101745) | 0.083639 / 0.424275 (-0.340636) | 0.012339 / 0.007607 (0.004732) | 0.498708 / 0.226044 (0.272663) | 4.995178 / 2.268929 (2.726249) | 2.272650 / 55.444624 (-53.171975) | 1.907879 / 6.876477 (-4.968598) | 2.012666 / 2.142072 (-0.129407) | 0.829564 / 4.805227 (-3.975663) | 0.165049 / 6.500664 (-6.335615) | 0.062291 / 0.075469 (-0.013178) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.193977 / 1.841788 (-0.647811) | 14.816939 / 8.074308 (6.742631) | 14.369729 / 10.191392 (4.178337) | 0.156339 / 0.680424 (-0.524084) | 0.029151 / 0.534201 (-0.505050) | 0.449362 / 0.579283 (-0.129921) | 0.451895 / 0.434364 (0.017531) | 0.520324 / 0.540337 (-0.020013) | 0.610716 / 1.386936 (-0.776220) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007145 / 0.011353 (-0.004207) | 0.005299 / 0.011008 (-0.005710) | 0.074216 / 0.038508 (0.035708) | 0.033015 / 0.023109 (0.009906) | 0.337117 / 0.275898 (0.061219) | 0.367161 / 0.323480 (0.043682) | 0.005898 / 0.007986 (-0.002088) | 0.005283 / 0.004328 (0.000955) | 0.073795 / 0.004250 (0.069544) | 0.049253 / 0.037052 (0.012201) | 0.343327 / 0.258489 (0.084838) | 0.396417 / 0.293841 (0.102576) | 0.037162 / 0.128546 (-0.091384) | 0.012456 / 0.075646 (-0.063191) | 0.086668 / 0.419271 (-0.332604) | 0.049937 / 0.043533 (0.006404) | 0.335138 / 0.255139 (0.079999) | 0.358111 / 0.283200 (0.074912) | 0.107328 / 0.141683 (-0.034355) | 1.482290 / 1.452155 (0.030135) | 1.557872 / 1.492716 (0.065156) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.343759 / 0.018006 (0.325752) | 0.542697 / 0.000490 (0.542207) | 0.025943 / 0.000200 (0.025743) | 0.000264 / 0.000054 (0.000209) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028469 / 0.037411 (-0.008943) | 0.108620 / 0.014526 (0.094094) | 0.123667 / 0.176557 (-0.052890) | 0.168829 / 0.737135 (-0.568306) | 0.125875 / 0.296338 (-0.170464) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.424640 / 0.215209 (0.209431) | 4.227611 / 2.077655 (2.149956) | 2.003605 / 1.504120 (0.499486) | 1.810696 / 1.541195 (0.269501) | 1.882700 / 1.468490 (0.414210) | 0.701361 / 4.584777 (-3.883416) | 3.808054 / 3.745712 (0.062342) | 3.234896 / 5.269862 (-2.034966) | 1.872195 / 4.565676 (-2.693482) | 0.088102 / 0.424275 (-0.336173) | 0.012810 / 0.007607 (0.005203) | 0.551855 / 0.226044 (0.325810) | 5.245654 / 2.268929 (2.976725) | 2.557123 / 55.444624 (-52.887502) | 2.238897 / 6.876477 (-4.637580) | 2.256260 / 2.142072 (0.114187) | 0.849804 / 4.805227 (-3.955424) | 0.170557 / 6.500664 (-6.330107) | 0.064718 / 0.075469 (-0.010751) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.271701 / 1.841788 (-0.570087) | 14.925010 / 8.074308 (6.850702) | 14.966948 / 10.191392 (4.775556) | 0.162966 / 0.680424 (-0.517458) | 0.017618 / 0.534201 (-0.516583) | 0.433484 / 0.579283 (-0.145799) | 0.430047 / 0.434364 (-0.004316) | 0.537356 / 0.540337 (-0.002981) | 0.639237 / 1.386936 (-0.747699) |

</details>
</details>

![](https://cml.dev/watermark.png#aba888cb4d225b1a05596f52258a079bda98df70 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.012054 / 0.011353 (0.000702) | 0.005923 / 0.011008 (-0.005085) | 0.129531 / 0.038508 (0.091023) | 0.036283 / 0.023109 (0.013173) | 0.374406 / 0.275898 (0.098508) | 0.452538 / 0.323480 (0.129058) | 0.009419 / 0.007986 (0.001434) | 0.004783 / 0.004328 (0.000454) | 0.095292 / 0.004250 (0.091042) | 0.041290 / 0.037052 (0.004238) | 0.403940 / 0.258489 (0.145451) | 0.443091 / 0.293841 (0.149250) | 0.054635 / 0.128546 (-0.073911) | 0.019062 / 0.075646 (-0.056584) | 0.417053 / 0.419271 (-0.002218) | 0.060865 / 0.043533 (0.017332) | 0.378535 / 0.255139 (0.123396) | 0.401036 / 0.283200 (0.117836) | 0.122959 / 0.141683 (-0.018724) | 1.768517 / 1.452155 (0.316362) | 1.794700 / 1.492716 (0.301984) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.246529 / 0.018006 (0.228523) | 0.576887 / 0.000490 (0.576397) | 0.005031 / 0.000200 (0.004831) | 0.000125 / 0.000054 (0.000070) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027363 / 0.037411 (-0.010049) | 0.119037 / 0.014526 (0.104511) | 0.148109 / 0.176557 (-0.028447) | 0.179370 / 0.737135 (-0.557765) | 0.145105 / 0.296338 (-0.151234) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.588748 / 0.215209 (0.373539) | 5.934433 / 2.077655 (3.856778) | 2.549811 / 1.504120 (1.045691) | 2.234616 / 1.541195 (0.693421) | 2.268002 / 1.468490 (0.799512) | 1.154643 / 4.584777 (-3.430134) | 5.333935 / 3.745712 (1.588223) | 2.971065 / 5.269862 (-2.298796) | 2.131427 / 4.565676 (-2.434250) | 0.127737 / 0.424275 (-0.296538) | 0.014699 / 0.007607 (0.007091) | 0.735160 / 0.226044 (0.509115) | 7.403838 / 2.268929 (5.134909) | 3.298169 / 55.444624 (-52.146455) | 2.661285 / 6.876477 (-4.215192) | 2.688877 / 2.142072 (0.546805) | 1.344110 / 4.805227 (-3.461118) | 0.242016 / 6.500664 (-6.258648) | 0.077418 / 0.075469 (0.001948) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.566426 / 1.841788 (-0.275362) | 17.144308 / 8.074308 (9.070000) | 19.360598 / 10.191392 (9.169206) | 0.238554 / 0.680424 (-0.441870) | 0.044946 / 0.534201 (-0.489255) | 0.554183 / 0.579283 (-0.025100) | 0.630175 / 0.434364 (0.195811) | 0.630319 / 0.540337 (0.089982) | 0.745060 / 1.386936 (-0.641876) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009255 / 0.011353 (-0.002098) | 0.006951 / 0.011008 (-0.004057) | 0.092021 / 0.038508 (0.053513) | 0.035588 / 0.023109 (0.012479) | 0.415564 / 0.275898 (0.139666) | 0.446393 / 0.323480 (0.122913) | 0.006532 / 0.007986 (-0.001453) | 0.005099 / 0.004328 (0.000771) | 0.094801 / 0.004250 (0.090550) | 0.044926 / 0.037052 (0.007874) | 0.439125 / 0.258489 (0.180636) | 0.473004 / 0.293841 (0.179163) | 0.057025 / 0.128546 (-0.071522) | 0.018711 / 0.075646 (-0.056935) | 0.110844 / 0.419271 (-0.308427) | 0.058347 / 0.043533 (0.014814) | 0.435721 / 0.255139 (0.180583) | 0.434624 / 0.283200 (0.151424) | 0.114505 / 0.141683 (-0.027178) | 1.722379 / 1.452155 (0.270225) | 1.775836 / 1.492716 (0.283120) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.275893 / 0.018006 (0.257887) | 0.552590 / 0.000490 (0.552100) | 0.007919 / 0.000200 (0.007719) | 0.000122 / 0.000054 (0.000068) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030003 / 0.037411 (-0.007408) | 0.130145 / 0.014526 (0.115619) | 0.131878 / 0.176557 (-0.044678) | 0.194693 / 0.737135 (-0.542442) | 0.137689 / 0.296338 (-0.158650) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.619591 / 0.215209 (0.404382) | 6.324095 / 2.077655 (4.246441) | 2.756563 / 1.504120 (1.252444) | 2.384744 / 1.541195 (0.843549) | 2.450407 / 1.468490 (0.981917) | 1.235391 / 4.584777 (-3.349386) | 5.535383 / 3.745712 (1.789671) | 4.831927 / 5.269862 (-0.437934) | 2.757158 / 4.565676 (-1.808519) | 0.133980 / 0.424275 (-0.290295) | 0.014965 / 0.007607 (0.007358) | 0.731423 / 0.226044 (0.505379) | 7.401850 / 2.268929 (5.132921) | 3.346585 / 55.444624 (-52.098039) | 2.705523 / 6.876477 (-4.170953) | 2.637397 / 2.142072 (0.495324) | 1.347745 / 4.805227 (-3.457482) | 0.248658 / 6.500664 (-6.252006) | 0.077427 / 0.075469 (0.001958) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.520860 / 1.841788 (-0.320928) | 17.153000 / 8.074308 (9.078692) | 19.051393 / 10.191392 (8.860001) | 0.236840 / 0.680424 (-0.443584) | 0.026638 / 0.534201 (-0.507563) | 0.518417 / 0.579283 (-0.060866) | 0.607555 / 0.434364 (0.173191) | 0.637381 / 0.540337 (0.097044) | 0.767109 / 1.386936 (-0.619827) |

</details>
</details>

![](https://cml.dev/watermark.png#5ee291f2c5e68a782c82f916e250d470a7e285e7 "CML watermark")


Great, I merged it. Thanks for the review :)

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006711 / 0.011353 (-0.004641) | 0.004472 / 0.011008 (-0.006536) | 0.099581 / 0.038508 (0.061073) | 0.028036 / 0.023109 (0.004927) | 0.301197 / 0.275898 (0.025298) | 0.339341 / 0.323480 (0.015861) | 0.005107 / 0.007986 (-0.002879) | 0.003312 / 0.004328 (-0.001017) | 0.075823 / 0.004250 (0.071573) | 0.040861 / 0.037052 (0.003809) | 0.303407 / 0.258489 (0.044918) | 0.350717 / 0.293841 (0.056876) | 0.031657 / 0.128546 (-0.096889) | 0.011627 / 0.075646 (-0.064020) | 0.325465 / 0.419271 (-0.093806) | 0.052671 / 0.043533 (0.009138) | 0.301953 / 0.255139 (0.046814) | 0.327164 / 0.283200 (0.043964) | 0.091264 / 0.141683 (-0.050419) | 1.508947 / 1.452155 (0.056792) | 1.605685 / 1.492716 (0.112968) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.202977 / 0.018006 (0.184971) | 0.400602 / 0.000490 (0.400112) | 0.003253 / 0.000200 (0.003053) | 0.000080 / 0.000054 (0.000025) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022453 / 0.037411 (-0.014958) | 0.098633 / 0.014526 (0.084107) | 0.105996 / 0.176557 (-0.070561) | 0.162428 / 0.737135 (-0.574707) | 0.107139 / 0.296338 (-0.189199) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.453061 / 0.215209 (0.237852) | 4.530844 / 2.077655 (2.453190) | 2.286394 / 1.504120 (0.782274) | 2.076479 / 1.541195 (0.535284) | 2.143730 / 1.468490 (0.675240) | 0.702540 / 4.584777 (-3.882237) | 3.442688 / 3.745712 (-0.303024) | 1.874429 / 5.269862 (-3.395433) | 1.172331 / 4.565676 (-3.393346) | 0.083643 / 0.424275 (-0.340632) | 0.012519 / 0.007607 (0.004911) | 0.556859 / 0.226044 (0.330814) | 5.582843 / 2.268929 (3.313915) | 2.753734 / 55.444624 (-52.690890) | 2.415771 / 6.876477 (-4.460705) | 2.531428 / 2.142072 (0.389356) | 0.813005 / 4.805227 (-3.992222) | 0.153322 / 6.500664 (-6.347343) | 0.068061 / 0.075469 (-0.007408) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.180481 / 1.841788 (-0.661306) | 13.623933 / 8.074308 (5.549625) | 14.431288 / 10.191392 (4.239896) | 0.127580 / 0.680424 (-0.552844) | 0.016714 / 0.534201 (-0.517487) | 0.394236 / 0.579283 (-0.185047) | 0.381718 / 0.434364 (-0.052646) | 0.486749 / 0.540337 (-0.053589) | 0.565939 / 1.386936 (-0.820997) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006720 / 0.011353 (-0.004633) | 0.004518 / 0.011008 (-0.006491) | 0.076819 / 0.038508 (0.038311) | 0.027272 / 0.023109 (0.004163) | 0.340890 / 0.275898 (0.064992) | 0.381435 / 0.323480 (0.057955) | 0.004980 / 0.007986 (-0.003005) | 0.003382 / 0.004328 (-0.000947) | 0.076368 / 0.004250 (0.072117) | 0.037365 / 0.037052 (0.000313) | 0.341484 / 0.258489 (0.082995) | 0.388917 / 0.293841 (0.095076) | 0.032004 / 0.128546 (-0.096543) | 0.011612 / 0.075646 (-0.064034) | 0.084929 / 0.419271 (-0.334342) | 0.041861 / 0.043533 (-0.001671) | 0.350392 / 0.255139 (0.095253) | 0.369745 / 0.283200 (0.086546) | 0.088301 / 0.141683 (-0.053382) | 1.587296 / 1.452155 (0.135141) | 1.629761 / 1.492716 (0.137045) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.174825 / 0.018006 (0.156818) | 0.414371 / 0.000490 (0.413881) | 0.001595 / 0.000200 (0.001395) | 0.000078 / 0.000054 (0.000023) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025403 / 0.037411 (-0.012009) | 0.099593 / 0.014526 (0.085067) | 0.108819 / 0.176557 (-0.067738) | 0.161613 / 0.737135 (-0.575523) | 0.112302 / 0.296338 (-0.184037) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.439234 / 0.215209 (0.224024) | 4.389073 / 2.077655 (2.311418) | 2.063215 / 1.504120 (0.559095) | 1.852550 / 1.541195 (0.311356) | 1.920014 / 1.468490 (0.451524) | 0.710255 / 4.584777 (-3.874522) | 3.430549 / 3.745712 (-0.315164) | 1.886072 / 5.269862 (-3.383790) | 1.177490 / 4.565676 (-3.388186) | 0.084877 / 0.424275 (-0.339398) | 0.012894 / 0.007607 (0.005287) | 0.544950 / 0.226044 (0.318906) | 5.467347 / 2.268929 (3.198419) | 2.508169 / 55.444624 (-52.936455) | 2.167756 / 6.876477 (-4.708721) | 2.212817 / 2.142072 (0.070744) | 0.824762 / 4.805227 (-3.980465) | 0.154387 / 6.500664 (-6.346277) | 0.068535 / 0.075469 (-0.006934) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.284165 / 1.841788 (-0.557623) | 14.153006 / 8.074308 (6.078697) | 14.152569 / 10.191392 (3.961177) | 0.130083 / 0.680424 (-0.550341) | 0.016556 / 0.534201 (-0.517645) | 0.383828 / 0.579283 (-0.195455) | 0.388241 / 0.434364 (-0.046123) | 0.477982 / 0.540337 (-0.062355) | 0.565583 / 1.386936 (-0.821353) |

</details>
</details>

![](https://cml.dev/watermark.png#f1e7442d34a059ff377437381542cc762feab057 "CML watermark")