Fix authentication issues
Opened by albertvillanova (closed)

This PR fixes 3 authentication issues:
- Fix authentication when passing `token`.
- Fix authentication in `Audio.decode_example` and `Image.decode_example`.
- Fix authentication to resolve `data_files` in repositories without script.

This PR also fixes our CI so that we properly test when passing `token` and we do not use the token stored in `HfFolder`.

Fix #6126.

## Details


### Fix authentication when passing `token`

See c0a77dc943de68a17f23f141517028c734c78623

The root issue was caused when the `token` was set in an already instantiated `DownloadConfig` and thus not propagated to `self._storage_options`:

  ```python
  download_config.token = token
  ```

As this usage pattern is very common, the fix consists in overriding `DownloadConfig.__setattr__`.

This fixes authentication issues in the following functions:
- `load_dataset` and `load_dataset_builder`
- `Dataset.push_to_hub` and `Dataset.push_to_hub`
- `inspect.get_dataset_config_info`, `inspect.get_dataset_infos` and `inspect.get_dataset_split_names`

### Fix authentication in `Audio.decode_example` and `Image.decode_example`. 

See: 58e62af004b6b8b84dcfd897a4bc71637cfa6c3f

The `token` was not set because the `repo_id` was wrongly tried to be parsed from an HTTP URL (`"http://..."`), instead of an HFFileSystem URL (`"hf://"`)

### Fix authentication to resolve `data_files` in repositories without script

See: e4684fc1032321abf0d494b0c130ea7c82ebda80

This is fixed by passing `download_config` to the function `create_builder_configs_from_metadata_configs`

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006103 / 0.011353 (-0.005250) | 0.003588 / 0.011008 (-0.007420) | 0.080335 / 0.038508 (0.041827) | 0.059634 / 0.023109 (0.036525) | 0.356093 / 0.275898 (0.080195) | 0.407376 / 0.323480 (0.083896) | 0.005343 / 0.007986 (-0.002643) | 0.002928 / 0.004328 (-0.001400) | 0.062580 / 0.004250 (0.058330) | 0.047544 / 0.037052 (0.010491) | 0.364305 / 0.258489 (0.105816) | 0.421463 / 0.293841 (0.127623) | 0.027249 / 0.128546 (-0.101298) | 0.008010 / 0.075646 (-0.067636) | 0.262543 / 0.419271 (-0.156728) | 0.044978 / 0.043533 (0.001445) | 0.339344 / 0.255139 (0.084205) | 0.395288 / 0.283200 (0.112088) | 0.021425 / 0.141683 (-0.120258) | 1.439767 / 1.452155 (-0.012387) | 1.498081 / 1.492716 (0.005365) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.196976 / 0.018006 (0.178970) | 0.435383 / 0.000490 (0.434893) | 0.004559 / 0.000200 (0.004359) | 0.000071 / 0.000054 (0.000016) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023653 / 0.037411 (-0.013759) | 0.072944 / 0.014526 (0.058418) | 0.083651 / 0.176557 (-0.092906) | 0.144590 / 0.737135 (-0.592545) | 0.084844 / 0.296338 (-0.211494) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.398752 / 0.215209 (0.183543) | 3.959539 / 2.077655 (1.881884) | 1.935277 / 1.504120 (0.431157) | 1.751994 / 1.541195 (0.210799) | 1.828386 / 1.468490 (0.359896) | 0.500492 / 4.584777 (-4.084284) | 3.086630 / 3.745712 (-0.659082) | 2.851664 / 5.269862 (-2.418198) | 1.869792 / 4.565676 (-2.695885) | 0.058509 / 0.424275 (-0.365766) | 0.006500 / 0.007607 (-0.001107) | 0.467468 / 0.226044 (0.241424) | 4.686168 / 2.268929 (2.417240) | 2.427632 / 55.444624 (-53.016993) | 2.193194 / 6.876477 (-4.683283) | 2.408574 / 2.142072 (0.266501) | 0.592173 / 4.805227 (-4.213054) | 0.125381 / 6.500664 (-6.375283) | 0.060679 / 0.075469 (-0.014790) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.236066 / 1.841788 (-0.605722) | 18.591689 / 8.074308 (10.517381) | 14.138774 / 10.191392 (3.947382) | 0.147455 / 0.680424 (-0.532968) | 0.016921 / 0.534201 (-0.517280) | 0.328129 / 0.579283 (-0.251154) | 0.348872 / 0.434364 (-0.085491) | 0.380311 / 0.540337 (-0.160026) | 0.532901 / 1.386936 (-0.854035) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005914 / 0.011353 (-0.005438) | 0.003614 / 0.011008 (-0.007394) | 0.062857 / 0.038508 (0.024349) | 0.060633 / 0.023109 (0.037524) | 0.419684 / 0.275898 (0.143786) | 0.449025 / 0.323480 (0.125546) | 0.004595 / 0.007986 (-0.003391) | 0.002861 / 0.004328 (-0.001467) | 0.063253 / 0.004250 (0.059003) | 0.048770 / 0.037052 (0.011718) | 0.419838 / 0.258489 (0.161349) | 0.465183 / 0.293841 (0.171342) | 0.027350 / 0.128546 (-0.101196) | 0.008065 / 0.075646 (-0.067582) | 0.068321 / 0.419271 (-0.350950) | 0.041083 / 0.043533 (-0.002449) | 0.400831 / 0.255139 (0.145692) | 0.449286 / 0.283200 (0.166086) | 0.020472 / 0.141683 (-0.121210) | 1.437215 / 1.452155 (-0.014940) | 1.503679 / 1.492716 (0.010963) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.230764 / 0.018006 (0.212758) | 0.420774 / 0.000490 (0.420285) | 0.004012 / 0.000200 (0.003812) | 0.000069 / 0.000054 (0.000014) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026009 / 0.037411 (-0.011402) | 0.077943 / 0.014526 (0.063417) | 0.087281 / 0.176557 (-0.089276) | 0.139422 / 0.737135 (-0.597713) | 0.089090 / 0.296338 (-0.207248) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.417298 / 0.215209 (0.202088) | 4.152303 / 2.077655 (2.074648) | 2.179996 / 1.504120 (0.675877) | 2.020619 / 1.541195 (0.479424) | 2.085241 / 1.468490 (0.616751) | 0.501111 / 4.584777 (-4.083666) | 3.079849 / 3.745712 (-0.665863) | 2.820607 / 5.269862 (-2.449255) | 1.863988 / 4.565676 (-2.701688) | 0.057662 / 0.424275 (-0.366613) | 0.006778 / 0.007607 (-0.000830) | 0.498661 / 0.226044 (0.272616) | 4.986503 / 2.268929 (2.717574) | 2.620676 / 55.444624 (-52.823949) | 2.297546 / 6.876477 (-4.578931) | 2.458148 / 2.142072 (0.316075) | 0.599490 / 4.805227 (-4.205738) | 0.125102 / 6.500664 (-6.375562) | 0.061411 / 0.075469 (-0.014059) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.323816 / 1.841788 (-0.517971) | 18.462614 / 8.074308 (10.388306) | 13.845826 / 10.191392 (3.654434) | 0.146115 / 0.680424 (-0.534309) | 0.016862 / 0.534201 (-0.517339) | 0.335449 / 0.579283 (-0.243834) | 0.343792 / 0.434364 (-0.090572) | 0.394068 / 0.540337 (-0.146269) | 0.536378 / 1.386936 (-0.850558) |

</details>
</details>

![](https://cml.dev/watermark.png#de3f00368c9236e9410821f5fddb95d6069883c1 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006825 / 0.011353 (-0.004527) | 0.004005 / 0.011008 (-0.007003) | 0.085504 / 0.038508 (0.046996) | 0.077252 / 0.023109 (0.054143) | 0.351891 / 0.275898 (0.075993) | 0.383404 / 0.323480 (0.059924) | 0.004153 / 0.007986 (-0.003833) | 0.003344 / 0.004328 (-0.000985) | 0.064936 / 0.004250 (0.060685) | 0.057653 / 0.037052 (0.020601) | 0.368155 / 0.258489 (0.109666) | 0.406122 / 0.293841 (0.112282) | 0.032049 / 0.128546 (-0.096497) | 0.008698 / 0.075646 (-0.066949) | 0.292394 / 0.419271 (-0.126878) | 0.053634 / 0.043533 (0.010101) | 0.358273 / 0.255139 (0.103134) | 0.378441 / 0.283200 (0.095242) | 0.026928 / 0.141683 (-0.114755) | 1.458718 / 1.452155 (0.006563) | 1.536231 / 1.492716 (0.043515) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.213956 / 0.018006 (0.195950) | 0.458620 / 0.000490 (0.458130) | 0.002718 / 0.000200 (0.002519) | 0.000078 / 0.000054 (0.000023) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027870 / 0.037411 (-0.009541) | 0.083922 / 0.014526 (0.069396) | 0.152056 / 0.176557 (-0.024501) | 0.151584 / 0.737135 (-0.585552) | 0.095698 / 0.296338 (-0.200641) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.407762 / 0.215209 (0.192553) | 4.074324 / 2.077655 (1.996669) | 2.089929 / 1.504120 (0.585809) | 1.920024 / 1.541195 (0.378829) | 2.013410 / 1.468490 (0.544920) | 0.486056 / 4.584777 (-4.098721) | 3.656869 / 3.745712 (-0.088843) | 3.304008 / 5.269862 (-1.965854) | 2.074363 / 4.565676 (-2.491313) | 0.057293 / 0.424275 (-0.366982) | 0.007240 / 0.007607 (-0.000367) | 0.482696 / 0.226044 (0.256652) | 4.833251 / 2.268929 (2.564322) | 2.570391 / 55.444624 (-52.874233) | 2.220619 / 6.876477 (-4.655857) | 2.426316 / 2.142072 (0.284243) | 0.584811 / 4.805227 (-4.220416) | 0.134907 / 6.500664 (-6.365757) | 0.061115 / 0.075469 (-0.014354) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.251969 / 1.841788 (-0.589818) | 19.601611 / 8.074308 (11.527303) | 14.190217 / 10.191392 (3.998825) | 0.166296 / 0.680424 (-0.514128) | 0.018334 / 0.534201 (-0.515867) | 0.395172 / 0.579283 (-0.184111) | 0.410440 / 0.434364 (-0.023924) | 0.462263 / 0.540337 (-0.078074) | 0.645504 / 1.386936 (-0.741432) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006991 / 0.011353 (-0.004362) | 0.004084 / 0.011008 (-0.006924) | 0.065208 / 0.038508 (0.026700) | 0.077809 / 0.023109 (0.054699) | 0.386472 / 0.275898 (0.110574) | 0.418686 / 0.323480 (0.095206) | 0.005346 / 0.007986 (-0.002640) | 0.003416 / 0.004328 (-0.000912) | 0.066209 / 0.004250 (0.061958) | 0.057517 / 0.037052 (0.020465) | 0.407684 / 0.258489 (0.149195) | 0.425438 / 0.293841 (0.131597) | 0.032166 / 0.128546 (-0.096380) | 0.008662 / 0.075646 (-0.066985) | 0.071712 / 0.419271 (-0.347560) | 0.049764 / 0.043533 (0.006231) | 0.394882 / 0.255139 (0.139743) | 0.403589 / 0.283200 (0.120389) | 0.023688 / 0.141683 (-0.117995) | 1.468488 / 1.452155 (0.016334) | 1.533118 / 1.492716 (0.040401) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.252949 / 0.018006 (0.234943) | 0.447355 / 0.000490 (0.446865) | 0.011721 / 0.000200 (0.011521) | 0.000107 / 0.000054 (0.000052) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031444 / 0.037411 (-0.005968) | 0.089390 / 0.014526 (0.074864) | 0.100103 / 0.176557 (-0.076454) | 0.153301 / 0.737135 (-0.583835) | 0.101336 / 0.296338 (-0.195003) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.408574 / 0.215209 (0.193365) | 4.073135 / 2.077655 (1.995480) | 2.086550 / 1.504120 (0.582430) | 1.930651 / 1.541195 (0.389457) | 2.013548 / 1.468490 (0.545058) | 0.477235 / 4.584777 (-4.107542) | 3.547545 / 3.745712 (-0.198167) | 3.321957 / 5.269862 (-1.947905) | 2.057705 / 4.565676 (-2.507971) | 0.056730 / 0.424275 (-0.367545) | 0.007882 / 0.007607 (0.000275) | 0.487297 / 0.226044 (0.261253) | 4.874184 / 2.268929 (2.605255) | 2.631129 / 55.444624 (-52.813496) | 2.235755 / 6.876477 (-4.640722) | 2.463329 / 2.142072 (0.321257) | 0.578308 / 4.805227 (-4.226919) | 0.132726 / 6.500664 (-6.367938) | 0.064883 / 0.075469 (-0.010586) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.347564 / 1.841788 (-0.494223) | 20.192973 / 8.074308 (12.118665) | 14.563553 / 10.191392 (4.372161) | 0.168244 / 0.680424 (-0.512180) | 0.018638 / 0.534201 (-0.515563) | 0.394789 / 0.579283 (-0.184494) | 0.419677 / 0.434364 (-0.014687) | 0.480274 / 0.540337 (-0.060063) | 0.641204 / 1.386936 (-0.745732) |

</details>
</details>

![](https://cml.dev/watermark.png#9c7a0d56b60bf700d6a491fa30eaf66500969315 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005939 / 0.011353 (-0.005413) | 0.003457 / 0.011008 (-0.007551) | 0.079985 / 0.038508 (0.041477) | 0.056492 / 0.023109 (0.033383) | 0.312356 / 0.275898 (0.036458) | 0.354038 / 0.323480 (0.030558) | 0.004551 / 0.007986 (-0.003435) | 0.002828 / 0.004328 (-0.001501) | 0.062369 / 0.004250 (0.058119) | 0.044712 / 0.037052 (0.007660) | 0.318244 / 0.258489 (0.059755) | 0.361977 / 0.293841 (0.068136) | 0.026460 / 0.128546 (-0.102086) | 0.007928 / 0.075646 (-0.067719) | 0.261378 / 0.419271 (-0.157894) | 0.044209 / 0.043533 (0.000676) | 0.313931 / 0.255139 (0.058792) | 0.339553 / 0.283200 (0.056354) | 0.019776 / 0.141683 (-0.121907) | 1.443126 / 1.452155 (-0.009029) | 1.508149 / 1.492716 (0.015432) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.183801 / 0.018006 (0.165795) | 0.427967 / 0.000490 (0.427477) | 0.002028 / 0.000200 (0.001828) | 0.000062 / 0.000054 (0.000007) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023697 / 0.037411 (-0.013715) | 0.072128 / 0.014526 (0.057602) | 0.083701 / 0.176557 (-0.092855) | 0.142821 / 0.737135 (-0.594315) | 0.082276 / 0.296338 (-0.214063) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.434427 / 0.215209 (0.219218) | 4.325962 / 2.077655 (2.248308) | 2.277115 / 1.504120 (0.772995) | 2.093736 / 1.541195 (0.552541) | 2.127984 / 1.468490 (0.659494) | 0.502336 / 4.584777 (-4.082441) | 3.023243 / 3.745712 (-0.722469) | 2.805154 / 5.269862 (-2.464708) | 1.821273 / 4.565676 (-2.744403) | 0.057480 / 0.424275 (-0.366795) | 0.006365 / 0.007607 (-0.001242) | 0.508258 / 0.226044 (0.282213) | 5.087950 / 2.268929 (2.819022) | 2.705029 / 55.444624 (-52.739596) | 2.378392 / 6.876477 (-4.498085) | 2.515380 / 2.142072 (0.373307) | 0.589283 / 4.805227 (-4.215944) | 0.125719 / 6.500664 (-6.374945) | 0.061074 / 0.075469 (-0.014395) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.221895 / 1.841788 (-0.619893) | 18.025917 / 8.074308 (9.951609) | 13.556901 / 10.191392 (3.365509) | 0.142614 / 0.680424 (-0.537809) | 0.016731 / 0.534201 (-0.517469) | 0.328374 / 0.579283 (-0.250910) | 0.342553 / 0.434364 (-0.091811) | 0.374502 / 0.540337 (-0.165836) | 0.534173 / 1.386936 (-0.852763) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005817 / 0.011353 (-0.005536) | 0.003500 / 0.011008 (-0.007509) | 0.062240 / 0.038508 (0.023732) | 0.058128 / 0.023109 (0.035019) | 0.424014 / 0.275898 (0.148116) | 0.468453 / 0.323480 (0.144973) | 0.004641 / 0.007986 (-0.003345) | 0.002821 / 0.004328 (-0.001508) | 0.062180 / 0.004250 (0.057930) | 0.047578 / 0.037052 (0.010526) | 0.427367 / 0.258489 (0.168878) | 0.467889 / 0.293841 (0.174048) | 0.027144 / 0.128546 (-0.101403) | 0.007969 / 0.075646 (-0.067678) | 0.067764 / 0.419271 (-0.351508) | 0.040719 / 0.043533 (-0.002814) | 0.423663 / 0.255139 (0.168524) | 0.458556 / 0.283200 (0.175356) | 0.019196 / 0.141683 (-0.122487) | 1.471546 / 1.452155 (0.019392) | 1.547541 / 1.492716 (0.054825) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.228777 / 0.018006 (0.210770) | 0.406663 / 0.000490 (0.406173) | 0.003688 / 0.000200 (0.003488) | 0.000075 / 0.000054 (0.000021) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025494 / 0.037411 (-0.011917) | 0.076339 / 0.014526 (0.061814) | 0.084233 / 0.176557 (-0.092324) | 0.136995 / 0.737135 (-0.600140) | 0.085443 / 0.296338 (-0.210895) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.420441 / 0.215209 (0.205232) | 4.187018 / 2.077655 (2.109363) | 2.142139 / 1.504120 (0.638019) | 1.974530 / 1.541195 (0.433335) | 2.027321 / 1.468490 (0.558831) | 0.498116 / 4.584777 (-4.086661) | 2.988514 / 3.745712 (-0.757198) | 2.782046 / 5.269862 (-2.487816) | 1.821725 / 4.565676 (-2.743951) | 0.057711 / 0.424275 (-0.366564) | 0.006664 / 0.007607 (-0.000944) | 0.491015 / 0.226044 (0.264971) | 4.921037 / 2.268929 (2.652108) | 2.574964 / 55.444624 (-52.869661) | 2.251703 / 6.876477 (-4.624774) | 2.361154 / 2.142072 (0.219082) | 0.593362 / 4.805227 (-4.211865) | 0.126107 / 6.500664 (-6.374557) | 0.061840 / 0.075469 (-0.013630) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.327459 / 1.841788 (-0.514328) | 18.062960 / 8.074308 (9.988652) | 13.669253 / 10.191392 (3.477861) | 0.130719 / 0.680424 (-0.549705) | 0.016564 / 0.534201 (-0.517637) | 0.335821 / 0.579283 (-0.243462) | 0.341691 / 0.434364 (-0.092673) | 0.392651 / 0.540337 (-0.147686) | 0.529650 / 1.386936 (-0.857286) |

</details>
</details>

![](https://cml.dev/watermark.png#c65806b0542996e56825ab46a3ce8f9c07ab0df3 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009625 / 0.011353 (-0.001728) | 0.005354 / 0.011008 (-0.005654) | 0.114350 / 0.038508 (0.075842) | 0.086637 / 0.023109 (0.063528) | 0.465381 / 0.275898 (0.189483) | 0.490411 / 0.323480 (0.166931) | 0.006575 / 0.007986 (-0.001411) | 0.004287 / 0.004328 (-0.000041) | 0.093134 / 0.004250 (0.088884) | 0.060209 / 0.037052 (0.023156) | 0.459570 / 0.258489 (0.201080) | 0.523320 / 0.293841 (0.229479) | 0.047943 / 0.128546 (-0.080603) | 0.014764 / 0.075646 (-0.060882) | 0.383887 / 0.419271 (-0.035384) | 0.069864 / 0.043533 (0.026331) | 0.469122 / 0.255139 (0.213983) | 0.509953 / 0.283200 (0.226753) | 0.037800 / 0.141683 (-0.103883) | 1.877589 / 1.452155 (0.425434) | 2.014913 / 1.492716 (0.522197) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.309146 / 0.018006 (0.291140) | 0.644390 / 0.000490 (0.643900) | 0.005017 / 0.000200 (0.004817) | 0.000102 / 0.000054 (0.000048) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032964 / 0.037411 (-0.004447) | 0.103236 / 0.014526 (0.088711) | 0.119950 / 0.176557 (-0.056607) | 0.207674 / 0.737135 (-0.529461) | 0.117278 / 0.296338 (-0.179060) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.605464 / 0.215209 (0.390255) | 6.027805 / 2.077655 (3.950150) | 2.719725 / 1.504120 (1.215605) | 2.262752 / 1.541195 (0.721558) | 2.330310 / 1.468490 (0.861820) | 0.862537 / 4.584777 (-3.722240) | 5.347080 / 3.745712 (1.601368) | 4.792170 / 5.269862 (-0.477691) | 3.103694 / 4.565676 (-1.461983) | 0.103646 / 0.424275 (-0.320629) | 0.009411 / 0.007607 (0.001804) | 0.743052 / 0.226044 (0.517008) | 7.289684 / 2.268929 (5.020755) | 3.436530 / 55.444624 (-52.008094) | 2.722440 / 6.876477 (-4.154036) | 2.952380 / 2.142072 (0.810308) | 1.047688 / 4.805227 (-3.757539) | 0.212724 / 6.500664 (-6.287940) | 0.081473 / 0.075469 (0.006004) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.714437 / 1.841788 (-0.127351) | 24.384330 / 8.074308 (16.310022) | 22.444162 / 10.191392 (12.252770) | 0.226264 / 0.680424 (-0.454160) | 0.030530 / 0.534201 (-0.503671) | 0.473999 / 0.579283 (-0.105284) | 0.575005 / 0.434364 (0.140641) | 0.542789 / 0.540337 (0.002451) | 0.776079 / 1.386936 (-0.610857) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009366 / 0.011353 (-0.001987) | 0.005239 / 0.011008 (-0.005769) | 0.085116 / 0.038508 (0.046608) | 0.089600 / 0.023109 (0.066491) | 0.485778 / 0.275898 (0.209880) | 0.540054 / 0.323480 (0.216574) | 0.006290 / 0.007986 (-0.001695) | 0.004054 / 0.004328 (-0.000274) | 0.083535 / 0.004250 (0.079284) | 0.067200 / 0.037052 (0.030148) | 0.519520 / 0.258489 (0.261031) | 0.544049 / 0.293841 (0.250208) | 0.054300 / 0.128546 (-0.074246) | 0.013650 / 0.075646 (-0.061996) | 0.102515 / 0.419271 (-0.316757) | 0.063054 / 0.043533 (0.019522) | 0.491724 / 0.255139 (0.236585) | 0.547498 / 0.283200 (0.264298) | 0.039266 / 0.141683 (-0.102416) | 1.801226 / 1.452155 (0.349071) | 1.861778 / 1.492716 (0.369061) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.313009 / 0.018006 (0.295003) | 0.587695 / 0.000490 (0.587205) | 0.004972 / 0.000200 (0.004772) | 0.000110 / 0.000054 (0.000055) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029230 / 0.037411 (-0.008181) | 0.091154 / 0.014526 (0.076628) | 0.110505 / 0.176557 (-0.066052) | 0.164204 / 0.737135 (-0.572932) | 0.107812 / 0.296338 (-0.188526) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.610535 / 0.215209 (0.395326) | 6.162517 / 2.077655 (4.084862) | 2.866718 / 1.504120 (1.362598) | 2.542412 / 1.541195 (1.001218) | 2.584136 / 1.468490 (1.115645) | 0.874319 / 4.584777 (-3.710458) | 5.257184 / 3.745712 (1.511472) | 4.705840 / 5.269862 (-0.564022) | 2.971708 / 4.565676 (-1.593969) | 0.099026 / 0.424275 (-0.325249) | 0.009142 / 0.007607 (0.001535) | 0.728660 / 0.226044 (0.502615) | 7.560922 / 2.268929 (5.291994) | 3.439521 / 55.444624 (-52.005103) | 2.854730 / 6.876477 (-4.021746) | 3.088951 / 2.142072 (0.946879) | 0.973621 / 4.805227 (-3.831606) | 0.209792 / 6.500664 (-6.290872) | 0.081107 / 0.075469 (0.005638) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.716809 / 1.841788 (-0.124978) | 24.386927 / 8.074308 (16.312619) | 20.715524 / 10.191392 (10.524131) | 0.260831 / 0.680424 (-0.419592) | 0.030701 / 0.534201 (-0.503500) | 0.490018 / 0.579283 (-0.089265) | 0.590424 / 0.434364 (0.156060) | 0.589942 / 0.540337 (0.049604) | 0.798094 / 1.386936 (-0.588842) |

</details>
</details>

![](https://cml.dev/watermark.png#c0a77dc943de68a17f23f141517028c734c78623 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006592 / 0.011353 (-0.004761) | 0.003880 / 0.011008 (-0.007128) | 0.083761 / 0.038508 (0.045253) | 0.075966 / 0.023109 (0.052857) | 0.315291 / 0.275898 (0.039393) | 0.355920 / 0.323480 (0.032440) | 0.004972 / 0.007986 (-0.003014) | 0.003053 / 0.004328 (-0.001275) | 0.063553 / 0.004250 (0.059302) | 0.050794 / 0.037052 (0.013742) | 0.317681 / 0.258489 (0.059192) | 0.361991 / 0.293841 (0.068150) | 0.028119 / 0.128546 (-0.100427) | 0.008203 / 0.075646 (-0.067443) | 0.271756 / 0.419271 (-0.147516) | 0.046701 / 0.043533 (0.003168) | 0.316520 / 0.255139 (0.061381) | 0.350499 / 0.283200 (0.067300) | 0.022399 / 0.141683 (-0.119284) | 1.416017 / 1.452155 (-0.036138) | 1.503087 / 1.492716 (0.010371) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.208250 / 0.018006 (0.190244) | 0.470345 / 0.000490 (0.469856) | 0.003687 / 0.000200 (0.003487) | 0.000073 / 0.000054 (0.000019) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026163 / 0.037411 (-0.011248) | 0.083315 / 0.014526 (0.068789) | 0.088541 / 0.176557 (-0.088015) | 0.150078 / 0.737135 (-0.587057) | 0.088862 / 0.296338 (-0.207476) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.404911 / 0.215209 (0.189702) | 4.059257 / 2.077655 (1.981602) | 1.890987 / 1.504120 (0.386867) | 1.726608 / 1.541195 (0.185413) | 1.767479 / 1.468490 (0.298989) | 0.518826 / 4.584777 (-4.065951) | 3.212145 / 3.745712 (-0.533567) | 3.029933 / 5.269862 (-2.239929) | 2.000203 / 4.565676 (-2.565474) | 0.059631 / 0.424275 (-0.364644) | 0.006707 / 0.007607 (-0.000900) | 0.485741 / 0.226044 (0.259697) | 4.871938 / 2.268929 (2.603010) | 2.418856 / 55.444624 (-53.025769) | 2.084847 / 6.876477 (-4.791630) | 2.207992 / 2.142072 (0.065920) | 0.614354 / 4.805227 (-4.190873) | 0.128932 / 6.500664 (-6.371732) | 0.062342 / 0.075469 (-0.013127) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.325792 / 1.841788 (-0.515995) | 19.718995 / 8.074308 (11.644687) | 15.278535 / 10.191392 (5.087143) | 0.146719 / 0.680424 (-0.533705) | 0.017718 / 0.534201 (-0.516483) | 0.335709 / 0.579283 (-0.243574) | 0.378060 / 0.434364 (-0.056304) | 0.391135 / 0.540337 (-0.149202) | 0.548045 / 1.386936 (-0.838891) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006504 / 0.011353 (-0.004849) | 0.003742 / 0.011008 (-0.007266) | 0.064405 / 0.038508 (0.025897) | 0.077618 / 0.023109 (0.054509) | 0.365325 / 0.275898 (0.089427) | 0.408109 / 0.323480 (0.084629) | 0.004909 / 0.007986 (-0.003076) | 0.002972 / 0.004328 (-0.001356) | 0.063933 / 0.004250 (0.059682) | 0.052916 / 0.037052 (0.015863) | 0.370891 / 0.258489 (0.112402) | 0.412134 / 0.293841 (0.118293) | 0.028171 / 0.128546 (-0.100375) | 0.008150 / 0.075646 (-0.067497) | 0.069248 / 0.419271 (-0.350024) | 0.042353 / 0.043533 (-0.001180) | 0.368117 / 0.255139 (0.112978) | 0.397548 / 0.283200 (0.114348) | 0.022967 / 0.141683 (-0.118716) | 1.472740 / 1.452155 (0.020586) | 1.524028 / 1.492716 (0.031311) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.256854 / 0.018006 (0.238848) | 0.471499 / 0.000490 (0.471009) | 0.009609 / 0.000200 (0.009409) | 0.000109 / 0.000054 (0.000054) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027978 / 0.037411 (-0.009433) | 0.086741 / 0.014526 (0.072215) | 0.091189 / 0.176557 (-0.085368) | 0.146117 / 0.737135 (-0.591018) | 0.092358 / 0.296338 (-0.203980) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.426356 / 0.215209 (0.211147) | 4.263782 / 2.077655 (2.186127) | 2.178198 / 1.504120 (0.674078) | 2.015405 / 1.541195 (0.474211) | 2.055966 / 1.468490 (0.587476) | 0.507531 / 4.584777 (-4.077246) | 3.175967 / 3.745712 (-0.569745) | 3.055697 / 5.269862 (-2.214165) | 1.987663 / 4.565676 (-2.578014) | 0.058452 / 0.424275 (-0.365823) | 0.006944 / 0.007607 (-0.000663) | 0.502534 / 0.226044 (0.276489) | 5.024693 / 2.268929 (2.755765) | 2.754971 / 55.444624 (-52.689653) | 2.470845 / 6.876477 (-4.405632) | 2.698675 / 2.142072 (0.556602) | 0.602357 / 4.805227 (-4.202871) | 0.129490 / 6.500664 (-6.371174) | 0.065127 / 0.075469 (-0.010342) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.398487 / 1.841788 (-0.443301) | 19.692279 / 8.074308 (11.617971) | 15.124064 / 10.191392 (4.932672) | 0.148938 / 0.680424 (-0.531486) | 0.017418 / 0.534201 (-0.516783) | 0.340480 / 0.579283 (-0.238803) | 0.377223 / 0.434364 (-0.057141) | 0.405303 / 0.540337 (-0.135034) | 0.548923 / 1.386936 (-0.838013) |

</details>
</details>

![](https://cml.dev/watermark.png#58e62af004b6b8b84dcfd897a4bc71637cfa6c3f "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006433 / 0.011353 (-0.004920) | 0.004002 / 0.011008 (-0.007006) | 0.084130 / 0.038508 (0.045622) | 0.070628 / 0.023109 (0.047519) | 0.312372 / 0.275898 (0.036474) | 0.343993 / 0.323480 (0.020513) | 0.003936 / 0.007986 (-0.004050) | 0.003336 / 0.004328 (-0.000993) | 0.064715 / 0.004250 (0.060465) | 0.052511 / 0.037052 (0.015458) | 0.314092 / 0.258489 (0.055603) | 0.363152 / 0.293841 (0.069311) | 0.030898 / 0.128546 (-0.097648) | 0.008396 / 0.075646 (-0.067250) | 0.288083 / 0.419271 (-0.131188) | 0.051654 / 0.043533 (0.008122) | 0.315252 / 0.255139 (0.060113) | 0.346756 / 0.283200 (0.063556) | 0.025167 / 0.141683 (-0.116515) | 1.487265 / 1.452155 (0.035110) | 1.557528 / 1.492716 (0.064812) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.206517 / 0.018006 (0.188510) | 0.458359 / 0.000490 (0.457869) | 0.003719 / 0.000200 (0.003519) | 0.000070 / 0.000054 (0.000016) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029631 / 0.037411 (-0.007780) | 0.083856 / 0.014526 (0.069330) | 0.340431 / 0.176557 (0.163875) | 0.153864 / 0.737135 (-0.583271) | 0.095951 / 0.296338 (-0.200388) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.379182 / 0.215209 (0.163973) | 3.783396 / 2.077655 (1.705741) | 1.835932 / 1.504120 (0.331813) | 1.667563 / 1.541195 (0.126369) | 1.739309 / 1.468490 (0.270818) | 0.478957 / 4.584777 (-4.105820) | 3.521974 / 3.745712 (-0.223738) | 3.237635 / 5.269862 (-2.032227) | 2.000300 / 4.565676 (-2.565377) | 0.056389 / 0.424275 (-0.367887) | 0.007242 / 0.007607 (-0.000365) | 0.452642 / 0.226044 (0.226598) | 4.524339 / 2.268929 (2.255411) | 2.346210 / 55.444624 (-53.098414) | 1.957196 / 6.876477 (-4.919281) | 2.180051 / 2.142072 (0.037979) | 0.570205 / 4.805227 (-4.235022) | 0.131346 / 6.500664 (-6.369318) | 0.059327 / 0.075469 (-0.016142) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.244709 / 1.841788 (-0.597079) | 19.566277 / 8.074308 (11.491969) | 14.172598 / 10.191392 (3.981206) | 0.166493 / 0.680424 (-0.513931) | 0.018281 / 0.534201 (-0.515920) | 0.391608 / 0.579283 (-0.187675) | 0.402642 / 0.434364 (-0.031722) | 0.464974 / 0.540337 (-0.075364) | 0.637565 / 1.386936 (-0.749371) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006929 / 0.011353 (-0.004424) | 0.004114 / 0.011008 (-0.006894) | 0.064589 / 0.038508 (0.026081) | 0.083334 / 0.023109 (0.060225) | 0.391280 / 0.275898 (0.115382) | 0.426157 / 0.323480 (0.102678) | 0.005336 / 0.007986 (-0.002650) | 0.003395 / 0.004328 (-0.000934) | 0.064560 / 0.004250 (0.060310) | 0.057094 / 0.037052 (0.020042) | 0.398959 / 0.258489 (0.140470) | 0.432470 / 0.293841 (0.138629) | 0.031412 / 0.128546 (-0.097134) | 0.008670 / 0.075646 (-0.066976) | 0.071249 / 0.419271 (-0.348022) | 0.048934 / 0.043533 (0.005401) | 0.384207 / 0.255139 (0.129068) | 0.407992 / 0.283200 (0.124792) | 0.024492 / 0.141683 (-0.117191) | 1.467788 / 1.452155 (0.015634) | 1.541011 / 1.492716 (0.048295) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.279607 / 0.018006 (0.261600) | 0.448899 / 0.000490 (0.448410) | 0.020990 / 0.000200 (0.020790) | 0.000132 / 0.000054 (0.000078) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030313 / 0.037411 (-0.007099) | 0.089209 / 0.014526 (0.074684) | 0.101024 / 0.176557 (-0.075532) | 0.153468 / 0.737135 (-0.583667) | 0.103219 / 0.296338 (-0.193120) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.429176 / 0.215209 (0.213967) | 4.302234 / 2.077655 (2.224580) | 2.291103 / 1.504120 (0.786983) | 2.126257 / 1.541195 (0.585062) | 2.207090 / 1.468490 (0.738600) | 0.484643 / 4.584777 (-4.100134) | 3.557429 / 3.745712 (-0.188283) | 3.253804 / 5.269862 (-2.016058) | 2.026087 / 4.565676 (-2.539589) | 0.057793 / 0.424275 (-0.366482) | 0.007761 / 0.007607 (0.000154) | 0.504819 / 0.226044 (0.278775) | 5.046868 / 2.268929 (2.777940) | 2.773149 / 55.444624 (-52.671475) | 2.398036 / 6.876477 (-4.478440) | 2.608094 / 2.142072 (0.466021) | 0.630499 / 4.805227 (-4.174729) | 0.135496 / 6.500664 (-6.365168) | 0.061329 / 0.075469 (-0.014140) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.327124 / 1.841788 (-0.514664) | 19.889796 / 8.074308 (11.815488) | 14.196100 / 10.191392 (4.004708) | 0.161963 / 0.680424 (-0.518461) | 0.018529 / 0.534201 (-0.515672) | 0.392325 / 0.579283 (-0.186958) | 0.404836 / 0.434364 (-0.029528) | 0.475898 / 0.540337 (-0.064439) | 0.633563 / 1.386936 (-0.753373) |

</details>
</details>

![](https://cml.dev/watermark.png#e4684fc1032321abf0d494b0c130ea7c82ebda80 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006390 / 0.011353 (-0.004963) | 0.003683 / 0.011008 (-0.007325) | 0.081274 / 0.038508 (0.042766) | 0.062193 / 0.023109 (0.039083) | 0.355360 / 0.275898 (0.079462) | 0.396471 / 0.323480 (0.072992) | 0.003569 / 0.007986 (-0.004416) | 0.003928 / 0.004328 (-0.000400) | 0.062292 / 0.004250 (0.058041) | 0.049700 / 0.037052 (0.012648) | 0.354604 / 0.258489 (0.096115) | 0.419436 / 0.293841 (0.125595) | 0.027151 / 0.128546 (-0.101395) | 0.007954 / 0.075646 (-0.067692) | 0.262231 / 0.419271 (-0.157041) | 0.045483 / 0.043533 (0.001950) | 0.354285 / 0.255139 (0.099146) | 0.385178 / 0.283200 (0.101978) | 0.021183 / 0.141683 (-0.120500) | 1.420785 / 1.452155 (-0.031370) | 1.531545 / 1.492716 (0.038829) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.202298 / 0.018006 (0.184292) | 0.442172 / 0.000490 (0.441683) | 0.003565 / 0.000200 (0.003366) | 0.000074 / 0.000054 (0.000020) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024229 / 0.037411 (-0.013183) | 0.074352 / 0.014526 (0.059826) | 0.087530 / 0.176557 (-0.089026) | 0.146478 / 0.737135 (-0.590658) | 0.085145 / 0.296338 (-0.211194) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.388395 / 0.215209 (0.173186) | 3.877623 / 2.077655 (1.799968) | 1.882444 / 1.504120 (0.378324) | 1.707871 / 1.541195 (0.166676) | 1.772132 / 1.468490 (0.303642) | 0.491937 / 4.584777 (-4.092840) | 3.057947 / 3.745712 (-0.687765) | 2.822390 / 5.269862 (-2.447471) | 1.879719 / 4.565676 (-2.685957) | 0.056830 / 0.424275 (-0.367445) | 0.006415 / 0.007607 (-0.001192) | 0.458945 / 0.226044 (0.232900) | 4.594502 / 2.268929 (2.325574) | 2.339677 / 55.444624 (-53.104948) | 1.983750 / 6.876477 (-4.892727) | 2.173792 / 2.142072 (0.031719) | 0.580390 / 4.805227 (-4.224838) | 0.124568 / 6.500664 (-6.376096) | 0.061694 / 0.075469 (-0.013775) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.265108 / 1.841788 (-0.576680) | 18.415254 / 8.074308 (10.340946) | 13.963829 / 10.191392 (3.772437) | 0.148926 / 0.680424 (-0.531498) | 0.016919 / 0.534201 (-0.517282) | 0.331082 / 0.579283 (-0.248201) | 0.345777 / 0.434364 (-0.088587) | 0.381123 / 0.540337 (-0.159214) | 0.543297 / 1.386936 (-0.843639) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006121 / 0.011353 (-0.005232) | 0.003717 / 0.011008 (-0.007291) | 0.063653 / 0.038508 (0.025144) | 0.063723 / 0.023109 (0.040613) | 0.360233 / 0.275898 (0.084335) | 0.398353 / 0.323480 (0.074873) | 0.004696 / 0.007986 (-0.003290) | 0.002876 / 0.004328 (-0.001452) | 0.063057 / 0.004250 (0.058806) | 0.050258 / 0.037052 (0.013206) | 0.362946 / 0.258489 (0.104457) | 0.403260 / 0.293841 (0.109419) | 0.027738 / 0.128546 (-0.100809) | 0.008025 / 0.075646 (-0.067621) | 0.068781 / 0.419271 (-0.350491) | 0.042114 / 0.043533 (-0.001419) | 0.363546 / 0.255139 (0.108407) | 0.385640 / 0.283200 (0.102440) | 0.021757 / 0.141683 (-0.119926) | 1.482364 / 1.452155 (0.030209) | 1.571859 / 1.492716 (0.079143) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.235628 / 0.018006 (0.217622) | 0.439909 / 0.000490 (0.439419) | 0.003070 / 0.000200 (0.002870) | 0.000075 / 0.000054 (0.000020) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027045 / 0.037411 (-0.010366) | 0.080413 / 0.014526 (0.065887) | 0.088953 / 0.176557 (-0.087603) | 0.141907 / 0.737135 (-0.595228) | 0.090604 / 0.296338 (-0.205735) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.423250 / 0.215209 (0.208041) | 4.216510 / 2.077655 (2.138855) | 2.162946 / 1.504120 (0.658826) | 2.014561 / 1.541195 (0.473366) | 2.086347 / 1.468490 (0.617857) | 0.496591 / 4.584777 (-4.088186) | 3.089594 / 3.745712 (-0.656118) | 2.853640 / 5.269862 (-2.416221) | 1.878149 / 4.565676 (-2.687527) | 0.056914 / 0.424275 (-0.367361) | 0.006762 / 0.007607 (-0.000845) | 0.493470 / 0.226044 (0.267426) | 4.929966 / 2.268929 (2.661037) | 2.640885 / 55.444624 (-52.803739) | 2.335950 / 6.876477 (-4.540527) | 2.565866 / 2.142072 (0.423793) | 0.585433 / 4.805227 (-4.219794) | 0.124969 / 6.500664 (-6.375695) | 0.062361 / 0.075469 (-0.013108) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.369144 / 1.841788 (-0.472644) | 19.037582 / 8.074308 (10.963274) | 14.069141 / 10.191392 (3.877749) | 0.146469 / 0.680424 (-0.533954) | 0.016911 / 0.534201 (-0.517290) | 0.336802 / 0.579283 (-0.242482) | 0.336411 / 0.434364 (-0.097953) | 0.392360 / 0.540337 (-0.147977) | 0.536078 / 1.386936 (-0.850858) |

</details>
</details>

![](https://cml.dev/watermark.png#12cfc1196e62847e2e8239fbd727a02cbc86ddec "CML watermark")